[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:18 dist-test-slave-m3pj krb5kdc[11145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:18 dist-test-slave-m3pj krb5kdc[11145](info): set up 2 sockets
Sep 26 12:38:18 dist-test-slave-m3pj krb5kdc[11145](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 12:38:19.156927 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:19 dist-test-slave-m3pj krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:19.296247 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34011
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:34011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:19.413779 11158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:19.414062 11158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:19.414101 11158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:19.418145 11158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:19.418208 11158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:19.418228 11158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:19.418279 11158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:19.418326 11158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:19.423732 11158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:34011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:19.424820 11158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:19.425880 11158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:38:19.435352 11158 server_base.cc:1034] running on GCE node
W20260926 12:38:19.435297 11163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:19.435305 11164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:19.436088 11166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:19.436692 11158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:19.437855 11158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:19.439028 11158 hybrid_clock.cc:648] HybridClock initialized: now 1790426299439007 us; error 27 us; skew 500 ppm
Sep 26 12:38:19 dist-test-slave-m3pj krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:19.449545 11158 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:19.450660 11158 webserver.cc:460] Webserver started at http://127.10.224.190:34751/ using document root <none> and password file <none>
I20260926 12:38:19.451279 11158 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:19.451344 11158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:19.451658 11158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:19.453848 11158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "0a7e8c1b76d3469fa79b18d91d0891f2"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "50705fa27a013b6aa340dfecf020c811"
server_key_iv: "3e50f3ace47548bab840acc6f98a995c"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.454424 11158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "0a7e8c1b76d3469fa79b18d91d0891f2"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "50705fa27a013b6aa340dfecf020c811"
server_key_iv: "3e50f3ace47548bab840acc6f98a995c"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.458372 11158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 12:38:19.461050 11173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:19.462386 11158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:38:19.462549 11158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "0a7e8c1b76d3469fa79b18d91d0891f2"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "50705fa27a013b6aa340dfecf020c811"
server_key_iv: "3e50f3ace47548bab840acc6f98a995c"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.462744 11158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:19.478724 11158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:19.482304 11158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:19.482534 11158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:19.490861 11158 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:34011
I20260926 12:38:19.490994 11225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:34011 every 8 connection(s)
I20260926 12:38:19.492049 11158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:19.494030 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 11158
I20260926 12:38:19.494215 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:19.495899 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:19.496165 11138 external_mini_cluster.cc:1428] Setting key 7a5a7588502b1140896af5c6da0ae23b
I20260926 12:38:19.503058 11226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2: Bootstrap starting.
I20260926 12:38:19.505897 11226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:19.506899 11226 log.cc:824] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:19.509276 11226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2: No bootstrap required, opened a new log
I20260926 12:38:19.514348 11226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7e8c1b76d3469fa79b18d91d0891f2" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34011 } }
I20260926 12:38:19.514635 11226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:19.514683 11226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7e8c1b76d3469fa79b18d91d0891f2, State: Initialized, Role: FOLLOWER
I20260926 12:38:19.515338 11226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7e8c1b76d3469fa79b18d91d0891f2" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34011 } }
I20260926 12:38:19.515511 11226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:19.515673 11226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:19.515849 11226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:19.516956 11226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7e8c1b76d3469fa79b18d91d0891f2" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34011 } }
I20260926 12:38:19.517473 11226 leader_election.cc:302] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a7e8c1b76d3469fa79b18d91d0891f2; no voters:
Sep 26 12:38:19 dist-test-slave-m3pj krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:19.517967 11226 leader_election.cc:288] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:19.518127 11231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:19.518400 11231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [term 1 LEADER]: Becoming Leader. State: Replica: 0a7e8c1b76d3469fa79b18d91d0891f2, State: Running, Role: LEADER
I20260926 12:38:19.518767 11231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7e8c1b76d3469fa79b18d91d0891f2" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34011 } }
I20260926 12:38:19.519688 11226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:19.520697 11232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a7e8c1b76d3469fa79b18d91d0891f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7e8c1b76d3469fa79b18d91d0891f2" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34011 } } }
I20260926 12:38:19.520843 11232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:19.521221 11233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a7e8c1b76d3469fa79b18d91d0891f2. Latest consensus state: current_term: 1 leader_uuid: "0a7e8c1b76d3469fa79b18d91d0891f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7e8c1b76d3469fa79b18d91d0891f2" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34011 } } }
I20260926 12:38:19.521335 11233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:19.521600 11240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:19.525976 11240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:19.527747 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:19.500429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.10.224.190:34011)
0926 12:38:19.500971 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:19.500991 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:19.502055 (+ 1064us) server_negotiation.cc:407] Connection header received
0926 12:38:19.505494 (+ 3439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:19.505517 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:19.505993 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:19.506409 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:19.508810 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:19.510955 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:19.512572 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:19.513459 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:19.513727 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:19.519233 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:19.519240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:19.519257 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:19.519301 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:19.521948 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:19.522782 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:19.522790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:19.522796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:19.522966 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:19.523405 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:19.523409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:19.523410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:19.524017 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:19.524220 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:19.526360 (+ 2140us) server_negotiation.cc:299] Negotiation successful
0926 12:38:19.526689 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":153,"threads_started":1}
I20260926 12:38:19.532590 11240 catalog_manager.cc:1347] Generated new cluster ID: f7e82a5b8a554e35a175a906878ba8e7
I20260926 12:38:19.532763 11240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:19.553761 11240 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:19.554831 11240 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:19.560031 11240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a7e8c1b76d3469fa79b18d91d0891f2: Generated new TSK 0
I20260926 12:38:19.560830 11240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:19.591375 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34011
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:19.710960 11252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:19.711350 11252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:19.711403 11252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:19.715875 11252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:19.715957 11252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:19.716023 11252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:19.722035 11252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:19.723177 11252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:19.724371 11252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:19.735635 11258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:19.735766 11257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:19.735635 11260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:19.736593 11252 server_base.cc:1034] running on GCE node
I20260926 12:38:19.737052 11252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:19.737741 11252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:19.738945 11252 hybrid_clock.cc:648] HybridClock initialized: now 1790426299738919 us; error 39 us; skew 500 ppm
Sep 26 12:38:19 dist-test-slave-m3pj krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:19.742158 11252 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:19.743201 11252 webserver.cc:460] Webserver started at http://127.10.224.129:45173/ using document root <none> and password file <none>
I20260926 12:38:19.743968 11252 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:19.744071 11252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:19.744328 11252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:19.746603 11252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "ae76a5d6d070467f955a2ca0cb14a84b"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "e9d0b158eac02de8278e0f090e8ebd19"
server_key_iv: "62189b3316e19ef662dd9f747320ffc3"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.747241 11252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "ae76a5d6d070467f955a2ca0cb14a84b"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "e9d0b158eac02de8278e0f090e8ebd19"
server_key_iv: "62189b3316e19ef662dd9f747320ffc3"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.751664 11252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:38:19.754467 11267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:19.756175 11252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:19.756340 11252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "ae76a5d6d070467f955a2ca0cb14a84b"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "e9d0b158eac02de8278e0f090e8ebd19"
server_key_iv: "62189b3316e19ef662dd9f747320ffc3"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.756491 11252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:19.768018 11252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:19.771329 11252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:19.771620 11252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:19.772248 11252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:19.773124 11252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:19.773195 11252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:19.773267 11252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:19.773319 11252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:19.782868 11252 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:46599
I20260926 12:38:19.782886 11380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:46599 every 8 connection(s)
I20260926 12:38:19.783955 11252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:19.788458 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 11252
I20260926 12:38:19.788688 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:19.788940 11138 external_mini_cluster.cc:1428] Setting key c3fa9b72c0ea07c20da4252324a49733
Sep 26 12:38:19 dist-test-slave-m3pj krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:38:19.799268 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:19.786128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:48627 (local address 127.10.224.190:34011)
0926 12:38:19.786272 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:19.786279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:19.787512 (+ 1233us) server_negotiation.cc:407] Connection header received
0926 12:38:19.788516 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:19.788519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:19.788579 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:19.788691 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:19.790232 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:19.790985 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:19.792343 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:19.792601 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:19.792768 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:19.795365 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:19.795369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:19.795372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:19.795412 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:19.797020 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:19.797565 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:19.797569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:19.797571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:19.797660 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:19.798044 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:19.798047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:19.798049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:19.798291 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:19.798381 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:19.798958 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 12:38:19.799094 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:38:19.800252 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:19.786539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34011 (local address 127.10.224.129:48627)
0926 12:38:19.787296 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:19.787356 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:19.788282 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:19.788893 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:19.788901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:19.789334 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:19.790057 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:19.790077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:19.791125 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:19.791128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:19.792203 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:19.792214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:19.792818 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:19.792823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:19.793008 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:19.793653 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:19.793671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:19.795178 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:19.797148 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:19.797154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:19.797173 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:19.797455 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:19.797766 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:19.797769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:19.797771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:19.797933 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:19.798395 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:19.798404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:19.798686 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 12:38:19.798918 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 12:38:19.799297 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:38:19.801790 11381 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34011
I20260926 12:38:19.802065 11381 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:19.802641 11381 heartbeater.cc:511] Master 127.10.224.190:34011 requested a full tablet report, sending...
I20260926 12:38:19.804533 11190 ts_manager.cc:194] Registered new tserver with Master: ae76a5d6d070467f955a2ca0cb14a84b (127.10.224.129:46599)
I20260926 12:38:19.806239 11190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:48627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:19.822942 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34011
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:19.940347 11386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:19.940626 11386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:19.940702 11386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:19.944888 11386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:19.944957 11386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:19.945060 11386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:19.950565 11386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:19.951730 11386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:19.952858 11386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:19.962889 11391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:19.962975 11392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:19.962935 11386 server_base.cc:1034] running on GCE node
W20260926 12:38:19.962955 11394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:19.963560 11386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:19.964285 11386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:19.965488 11386 hybrid_clock.cc:648] HybridClock initialized: now 1790426299965442 us; error 55 us; skew 500 ppm
Sep 26 12:38:19 dist-test-slave-m3pj krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:19.968497 11386 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:19.969444 11386 webserver.cc:460] Webserver started at http://127.10.224.130:38269/ using document root <none> and password file <none>
I20260926 12:38:19.970003 11386 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:19.970063 11386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:19.970242 11386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:19.972409 11386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "9f4e8e0fdf734f8e9ea60bd956de5010"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "f9295e8b70bc3e521cf39485913e9279"
server_key_iv: "6644a9ac29e7f99801f51aa26a611e4d"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.972961 11386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "9f4e8e0fdf734f8e9ea60bd956de5010"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "f9295e8b70bc3e521cf39485913e9279"
server_key_iv: "6644a9ac29e7f99801f51aa26a611e4d"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.977006 11386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:19.979959 11401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:19.981420 11386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:38:19.981595 11386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "9f4e8e0fdf734f8e9ea60bd956de5010"
format_stamp: "Formatted at 2026-09-26 12:38:19 on dist-test-slave-m3pj"
server_key: "f9295e8b70bc3e521cf39485913e9279"
server_key_iv: "6644a9ac29e7f99801f51aa26a611e4d"
server_key_version: "encryptionkey@0"
I20260926 12:38:19.981748 11386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:20.014709 11386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:20.018383 11386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:20.018611 11386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:20.019270 11386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:20.020318 11386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:20.020367 11386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:20.020411 11386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:20.020467 11386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:20.030505 11386 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:42503
I20260926 12:38:20.030522 11514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:42503 every 8 connection(s)
I20260926 12:38:20.031498 11386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:20.040561 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 11386
I20260926 12:38:20.040666 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:20.040941 11138 external_mini_cluster.cc:1428] Setting key d30374a15a96147836d9beafbb14b853
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:20.047833 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:20.033725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:45061 (local address 127.10.224.190:34011)
0926 12:38:20.033879 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:20.033883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:20.035120 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 12:38:20.036328 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:20.036332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:20.036397 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:20.036520 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:20.037903 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.038484 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.039886 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.040140 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.040270 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:20.043601 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:20.043609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:20.043612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:20.043651 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:20.045416 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:20.046041 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:20.046050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:20.046052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:20.046137 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:20.046547 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:20.046552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:20.046554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:20.046763 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:20.046898 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:20.047480 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 12:38:20.047649 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:38:20.048832 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:20.034162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34011 (local address 127.10.224.130:45061)
0926 12:38:20.034934 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:20.034981 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:20.036065 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:20.036716 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:20.036724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:20.037157 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:20.037722 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:20.037738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.038630 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.038642 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:20.039726 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:20.039737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.040318 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.040322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:20.040533 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:20.041318 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:20.041378 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:20.043356 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:20.045562 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:20.045569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:20.045589 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:20.045907 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:20.046242 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:20.046247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:20.046249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:20.046430 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:20.046886 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:20.046894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:20.047202 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 12:38:20.047436 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 12:38:20.047863 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":135,"threads_started":1}
I20260926 12:38:20.050194 11515 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34011
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:38:20.050526 11515 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:20.051128 11515 heartbeater.cc:511] Master 127.10.224.190:34011 requested a full tablet report, sending...
I20260926 12:38:20.052474 11190 ts_manager.cc:194] Registered new tserver with Master: 9f4e8e0fdf734f8e9ea60bd956de5010 (127.10.224.130:42503)
I20260926 12:38:20.053103 11190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:45061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:20.072836 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34011
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:20.189438 11520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:20.189760 11520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:20.189865 11520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:20.194157 11520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:20.194253 11520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:20.194347 11520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:20.200577 11520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:20.201810 11520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:20.202881 11520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:20.212951 11528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:20.213035 11526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:20.213007 11525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:20.213001 11520 server_base.cc:1034] running on GCE node
I20260926 12:38:20.213574 11520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:20.214272 11520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:20.215445 11520 hybrid_clock.cc:648] HybridClock initialized: now 1790426300215428 us; error 31 us; skew 500 ppm
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426300, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:20.218721 11520 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:20.219944 11520 webserver.cc:460] Webserver started at http://127.10.224.131:36239/ using document root <none> and password file <none>
I20260926 12:38:20.220645 11520 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:20.220716 11520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:20.221020 11520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:20.223438 11520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "66e885b84c7544ed95f3030dc8e7c555"
format_stamp: "Formatted at 2026-09-26 12:38:20 on dist-test-slave-m3pj"
server_key: "7a067f38eaca1f260cee502727e61b63"
server_key_iv: "2d0d47eb017ed8e79c79619ba3c604a9"
server_key_version: "encryptionkey@0"
I20260926 12:38:20.224087 11520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "66e885b84c7544ed95f3030dc8e7c555"
format_stamp: "Formatted at 2026-09-26 12:38:20 on dist-test-slave-m3pj"
server_key: "7a067f38eaca1f260cee502727e61b63"
server_key_iv: "2d0d47eb017ed8e79c79619ba3c604a9"
server_key_version: "encryptionkey@0"
I20260926 12:38:20.228123 11520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:20.230983 11535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:20.232492 11520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:20.232650 11520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "66e885b84c7544ed95f3030dc8e7c555"
format_stamp: "Formatted at 2026-09-26 12:38:20 on dist-test-slave-m3pj"
server_key: "7a067f38eaca1f260cee502727e61b63"
server_key_iv: "2d0d47eb017ed8e79c79619ba3c604a9"
server_key_version: "encryptionkey@0"
I20260926 12:38:20.232767 11520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:20.253752 11520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:20.257601 11520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:20.257776 11520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:20.258347 11520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:20.259199 11520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:20.259248 11520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:20.259289 11520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:20.259305 11520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:20.268716 11520 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:38039
I20260926 12:38:20.268734 11648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:38039 every 8 connection(s)
I20260926 12:38:20.269780 11520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426300, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:20.280102 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 11520
I20260926 12:38:20.280227 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:20.280498 11138 external_mini_cluster.cc:1428] Setting key 502c5512c0e0350c26c47a0d0dcc3149
I20260926 12:38:20.284514 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:20.271908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:52635 (local address 127.10.224.190:34011)
0926 12:38:20.272060 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:20.272064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:20.273368 (+ 1304us) server_negotiation.cc:407] Connection header received
0926 12:38:20.274290 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:20.274293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:20.274356 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:20.274443 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:20.275734 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.276277 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.277488 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.277725 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.277845 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:20.280615 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:20.280619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:20.280621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:20.280646 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:20.282358 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:20.282937 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:20.282942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:20.282944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:20.283038 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:20.283439 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:20.283441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:20.283443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:20.283699 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:20.283780 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:20.284287 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 12:38:20.284394 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:38:20.285451 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:20.272324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34011 (local address 127.10.224.131:52635)
0926 12:38:20.273213 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:20.273257 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:20.274085 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:20.274661 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:20.274668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:20.275027 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:20.275575 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:20.275602 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.276438 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.276441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:20.277349 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:20.277358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.277887 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.277891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:20.278077 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:20.278723 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:20.278740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:20.280394 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:20.282507 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:20.282514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:20.282531 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:20.282825 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:20.283135 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:20.283139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:20.283142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:20.283335 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:20.283792 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:20.283798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:20.284075 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 12:38:20.284242 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 12:38:20.284518 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":541,"threads_started":1}
I20260926 12:38:20.286729 11649 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34011
I20260926 12:38:20.286995 11649 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:20.287626 11649 heartbeater.cc:511] Master 127.10.224.190:34011 requested a full tablet report, sending...
I20260926 12:38:20.288789 11190 ts_manager.cc:194] Registered new tserver with Master: 66e885b84c7544ed95f3030dc8e7c555 (127.10.224.131:38039)
I20260926 12:38:20.289350 11190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:52635
I20260926 12:38:20.295491 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:20.323103 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:20.312595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.10.224.129:46599)
0926 12:38:20.312989 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:20.312996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:20.313024 (+ 28us) server_negotiation.cc:407] Connection header received
0926 12:38:20.313081 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:20.313087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:20.313293 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:20.313432 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:20.314116 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.314981 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.315934 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.316357 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.316466 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:20.319178 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:20.319195 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:20.319212 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:20.319241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:20.321110 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:20.321754 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:20.321760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:20.321761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:20.321869 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:20.322143 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:20.322146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:20.322147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:20.322439 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:20.322636 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:20.322818 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 12:38:20.322957 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
W20260926 12:38:20.324249 11295 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47648
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:20.351962 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:20.342861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.10.224.129:46599)
0926 12:38:20.343020 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:20.343025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:20.343219 (+ 194us) server_negotiation.cc:407] Connection header received
0926 12:38:20.343307 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:20.343310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:20.343357 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:20.343466 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:20.344294 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.344865 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.345675 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:20.345894 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:20.345977 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:20.348570 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:20.348573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:20.348575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:20.348604 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:20.349933 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:20.350583 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:20.350587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:20.350588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:20.350677 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:20.351013 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:20.351016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:20.351018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:20.351207 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:20.351398 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:20.351647 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 12:38:20.351760 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:38:20.353367 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 11252
I20260926 12:38:20.359884 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 11386
I20260926 12:38:20.368327 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 11520
I20260926 12:38:20.374240 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 11158
2026-09-26T12:38:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2287 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11663](info): set up 2 sockets
Sep 26 12:38:20 dist-test-slave-m3pj krb5kdc[11663](info): commencing operation
krb5kdc: starting...
W20260926 12:38:21.419783 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:21 dist-test-slave-m3pj krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:21.551052 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:46419
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:46419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:21.668620 11676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:21.668920 11676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:21.668963 11676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:21.673344 11676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:21.673429 11676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:21.673449 11676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:21.673466 11676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:21.673483 11676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:21.679279 11676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:46419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:46419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:21.680533 11676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:21.681696 11676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:21.690935 11682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:21.691041 11676 server_base.cc:1034] running on GCE node
W20260926 12:38:21.690935 11681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:21.691248 11684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:21.691885 11676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:21.693104 11676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:21.694272 11676 hybrid_clock.cc:648] HybridClock initialized: now 1790426301694228 us; error 58 us; skew 500 ppm
Sep 26 12:38:21 dist-test-slave-m3pj krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426301, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:21.697319 11676 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:21.698446 11676 webserver.cc:460] Webserver started at http://127.10.224.190:40317/ using document root <none> and password file <none>
I20260926 12:38:21.699071 11676 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:21.699136 11676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:21.699378 11676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:21.701725 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "962efdabf55049b5987d26373cb09ab3"
format_stamp: "Formatted at 2026-09-26 12:38:21 on dist-test-slave-m3pj"
server_key: "a1726c3b354749bbf6c5386346af0024"
server_key_iv: "f2c339a13489f293bcbdcae882ddae4e"
server_key_version: "encryptionkey@0"
I20260926 12:38:21.702368 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "962efdabf55049b5987d26373cb09ab3"
format_stamp: "Formatted at 2026-09-26 12:38:21 on dist-test-slave-m3pj"
server_key: "a1726c3b354749bbf6c5386346af0024"
server_key_iv: "f2c339a13489f293bcbdcae882ddae4e"
server_key_version: "encryptionkey@0"
I20260926 12:38:21.706450 11676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:21.709080 11691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:21.710407 11676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:38:21.710542 11676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "962efdabf55049b5987d26373cb09ab3"
format_stamp: "Formatted at 2026-09-26 12:38:21 on dist-test-slave-m3pj"
server_key: "a1726c3b354749bbf6c5386346af0024"
server_key_iv: "f2c339a13489f293bcbdcae882ddae4e"
server_key_version: "encryptionkey@0"
I20260926 12:38:21.710660 11676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:21.728077 11676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:21.731758 11676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:21.731997 11676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:21.739679 11676 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:46419
I20260926 12:38:21.739694 11743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:46419 every 8 connection(s)
I20260926 12:38:21.740814 11676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:21.744032 11744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:21.748507 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 11676
I20260926 12:38:21.748621 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:21.748926 11138 external_mini_cluster.cc:1428] Setting key 8b5846111f6d6391dcef12496c852a0e
I20260926 12:38:21.751124 11744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3: Bootstrap starting.
I20260926 12:38:21.754071 11744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:21.754915 11744 log.cc:824] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:21.757679 11744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3: No bootstrap required, opened a new log
Sep 26 12:38:21 dist-test-slave-m3pj krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:21.763068 11744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962efdabf55049b5987d26373cb09ab3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46419 } }
I20260926 12:38:21.763448 11744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:21.763545 11744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962efdabf55049b5987d26373cb09ab3, State: Initialized, Role: FOLLOWER
I20260926 12:38:21.764142 11744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962efdabf55049b5987d26373cb09ab3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46419 } }
I20260926 12:38:21.764341 11744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:21.764428 11744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:21.764541 11744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:21.765838 11744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962efdabf55049b5987d26373cb09ab3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46419 } }
I20260926 12:38:21.766239 11744 leader_election.cc:302] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 962efdabf55049b5987d26373cb09ab3; no voters:
I20260926 12:38:21.766620 11744 leader_election.cc:288] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:21.766680 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:21.750781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.10.224.190:46419)
0926 12:38:21.751289 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:21.751313 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:21.751376 (+ 63us) server_negotiation.cc:407] Connection header received
0926 12:38:21.752183 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:21.752203 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:21.752605 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:21.752937 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:21.753937 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:21.755262 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:21.756674 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:21.757248 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:21.757393 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:21.760609 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:21.760616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:21.760639 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:21.760681 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:21.763013 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:21.763638 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:21.763646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:21.763653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:21.763809 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:21.764132 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:21.764135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:21.764137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:21.764789 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:21.765012 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:21.765242 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 12:38:21.765580 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":168,"threads_started":1}
I20260926 12:38:21.766769 11749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:21.767055 11749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [term 1 LEADER]: Becoming Leader. State: Replica: 962efdabf55049b5987d26373cb09ab3, State: Running, Role: LEADER
I20260926 12:38:21.767463 11749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962efdabf55049b5987d26373cb09ab3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46419 } }
I20260926 12:38:21.768026 11744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:21.768828 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 962efdabf55049b5987d26373cb09ab3. Latest consensus state: current_term: 1 leader_uuid: "962efdabf55049b5987d26373cb09ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962efdabf55049b5987d26373cb09ab3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46419 } } }
I20260926 12:38:21.768949 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:21.769795 11758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:21.769909 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "962efdabf55049b5987d26373cb09ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962efdabf55049b5987d26373cb09ab3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46419 } } }
I20260926 12:38:21.769996 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:21.773001 11758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:21.779446 11758 catalog_manager.cc:1347] Generated new cluster ID: 84b85486f2ae423e90518f2521beeeef
I20260926 12:38:21.779573 11758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:21.787204 11758 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:21.788604 11758 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:21.798097 11758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 962efdabf55049b5987d26373cb09ab3: Generated new TSK 0
I20260926 12:38:21.799053 11758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:21.831207 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46419
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:21.952250 11770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:21.952584 11770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:21.952633 11770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:21.957098 11770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:21.957206 11770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:21.957322 11770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:21.963037 11770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:21.964264 11770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:21.965379 11770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:21.975956 11775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:21.975975 11778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:21.976019 11770 server_base.cc:1034] running on GCE node
W20260926 12:38:21.975983 11776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:21.976714 11770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:21.977463 11770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:21.978677 11770 hybrid_clock.cc:648] HybridClock initialized: now 1790426301978657 us; error 30 us; skew 500 ppm
Sep 26 12:38:21 dist-test-slave-m3pj krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426301, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:21.981842 11770 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:21.982964 11770 webserver.cc:460] Webserver started at http://127.10.224.129:40561/ using document root <none> and password file <none>
I20260926 12:38:21.983630 11770 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:21.983704 11770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:21.983940 11770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:21.986169 11770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "1b30bb27623a4d09a89fd2548a733d95"
format_stamp: "Formatted at 2026-09-26 12:38:21 on dist-test-slave-m3pj"
server_key: "5696ce431359d7d5bc2701dd4eb87916"
server_key_iv: "25f795fc5f3c92819bfbc274c41ff6f5"
server_key_version: "encryptionkey@0"
I20260926 12:38:21.986814 11770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "1b30bb27623a4d09a89fd2548a733d95"
format_stamp: "Formatted at 2026-09-26 12:38:21 on dist-test-slave-m3pj"
server_key: "5696ce431359d7d5bc2701dd4eb87916"
server_key_iv: "25f795fc5f3c92819bfbc274c41ff6f5"
server_key_version: "encryptionkey@0"
I20260926 12:38:21.991205 11770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:21.994095 11785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:21.995819 11770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:21.995970 11770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "1b30bb27623a4d09a89fd2548a733d95"
format_stamp: "Formatted at 2026-09-26 12:38:21 on dist-test-slave-m3pj"
server_key: "5696ce431359d7d5bc2701dd4eb87916"
server_key_iv: "25f795fc5f3c92819bfbc274c41ff6f5"
server_key_version: "encryptionkey@0"
I20260926 12:38:21.996100 11770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:22.040257 11770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:22.044070 11770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:22.044312 11770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:22.044986 11770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:22.045909 11770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:22.045962 11770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:22.046010 11770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:22.046069 11770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:22.055459 11770 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:46635
I20260926 12:38:22.055486 11898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:46635 every 8 connection(s)
I20260926 12:38:22.056499 11770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:22.058889 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 11770
I20260926 12:38:22.058976 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:22.059230 11138 external_mini_cluster.cc:1428] Setting key 7cbce4693973fdff960d2bf76492533c
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426301, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:38:22.073271 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.058506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:51431 (local address 127.10.224.190:46419)
0926 12:38:22.058693 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.058699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.059815 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 12:38:22.061573 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.061576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.061626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.061699 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:22.063236 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.064015 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.065288 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.065674 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.065761 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.069230 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:22.069233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:22.069235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:22.069263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:22.070988 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.071596 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.071600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.071602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.071693 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.072074 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.072078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.072080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.072292 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:22.072404 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.072953 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.073097 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:38:22.074200 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.059009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:46419 (local address 127.10.224.129:51431)
0926 12:38:22.059624 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:22.059686 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:22.061314 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:22.061934 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:22.061947 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:22.062510 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:22.063085 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.063099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.064166 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.064169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.065158 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.065168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.065858 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.065865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.066133 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.066873 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:22.066903 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:22.069009 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:22.071137 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:22.071145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:22.071164 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:22.071463 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:22.071791 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:22.071795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:22.071797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:22.071965 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:22.072403 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:22.072412 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:22.072698 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 12:38:22.072902 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 12:38:22.073285 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":169,"threads_started":1}
I20260926 12:38:22.075743 11899 heartbeater.cc:347] Connected to a master server at 127.10.224.190:46419
I20260926 12:38:22.076062 11899 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:22.076716 11899 heartbeater.cc:511] Master 127.10.224.190:46419 requested a full tablet report, sending...
I20260926 12:38:22.078536 11708 ts_manager.cc:194] Registered new tserver with Master: 1b30bb27623a4d09a89fd2548a733d95 (127.10.224.129:46635)
I20260926 12:38:22.079821 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:51431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:22.091223 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46419
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:22.210180 11904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:22.210481 11904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:22.210530 11904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:22.215019 11904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:22.215142 11904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:22.215251 11904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:22.221235 11904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:22.222493 11904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:22.223660 11904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:22.234432 11909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:22.234408 11910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:22.234416 11912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:22.234791 11904 server_base.cc:1034] running on GCE node
I20260926 12:38:22.235347 11904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:22.236158 11904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:22.237378 11904 hybrid_clock.cc:648] HybridClock initialized: now 1790426302237360 us; error 34 us; skew 500 ppm
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:22.240640 11904 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:22.241638 11904 webserver.cc:460] Webserver started at http://127.10.224.130:40117/ using document root <none> and password file <none>
I20260926 12:38:22.242236 11904 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:22.242302 11904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:22.242491 11904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:22.244871 11904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "fbc8217c23d7450fb498401da2826d23"
format_stamp: "Formatted at 2026-09-26 12:38:22 on dist-test-slave-m3pj"
server_key: "6f2ef2baa3f9059f89d5975d1ae45011"
server_key_iv: "f9d5be77e1ad73aa00633705b5b08f00"
server_key_version: "encryptionkey@0"
I20260926 12:38:22.245469 11904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "fbc8217c23d7450fb498401da2826d23"
format_stamp: "Formatted at 2026-09-26 12:38:22 on dist-test-slave-m3pj"
server_key: "6f2ef2baa3f9059f89d5975d1ae45011"
server_key_iv: "f9d5be77e1ad73aa00633705b5b08f00"
server_key_version: "encryptionkey@0"
I20260926 12:38:22.249660 11904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:22.252552 11919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:22.254027 11904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:22.254144 11904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "fbc8217c23d7450fb498401da2826d23"
format_stamp: "Formatted at 2026-09-26 12:38:22 on dist-test-slave-m3pj"
server_key: "6f2ef2baa3f9059f89d5975d1ae45011"
server_key_iv: "f9d5be77e1ad73aa00633705b5b08f00"
server_key_version: "encryptionkey@0"
I20260926 12:38:22.254251 11904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:22.282768 11904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:22.286523 11904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:22.286695 11904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:22.287282 11904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:22.288280 11904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:22.288328 11904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:22.288403 11904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:22.288434 11904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:22.298797 11904 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:38027
I20260926 12:38:22.298818 12032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:38027 every 8 connection(s)
I20260926 12:38:22.300029 11904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:22.308753 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 11904
I20260926 12:38:22.308903 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:22.309177 11138 external_mini_cluster.cc:1428] Setting key 4504d89089d32fb5a3ffbd7730ce7a3b
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:22.316740 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.302214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:54801 (local address 127.10.224.190:46419)
0926 12:38:22.302377 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.302381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.303328 (+ 947us) server_negotiation.cc:407] Connection header received
0926 12:38:22.304983 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.304987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.305048 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.305172 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:22.306514 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.307108 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.308475 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.308736 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.308886 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.311949 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:22.311953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:22.311956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:22.311982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:22.313792 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.314398 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.314402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.314404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.314508 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.315000 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.315007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.315011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.315317 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:22.315481 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.316473 (+ 992us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.316605 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 12:38:22.318295 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.302652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:46419 (local address 127.10.224.130:54801)
0926 12:38:22.303153 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:22.303203 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:22.304739 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:22.305388 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:22.305396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:22.305803 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:22.306348 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.306364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.307256 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.307260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.308333 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.308343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.308947 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.308952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.309134 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.309880 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:22.309899 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:22.311743 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:22.313925 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:22.313933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:22.313954 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:22.314267 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:22.314613 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:22.314618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:22.314620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:22.314858 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:22.315623 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:22.315635 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:22.316100 (+ 465us) client_negotiation.cc:769] Sending connection context
0926 12:38:22.316389 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 12:38:22.316859 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260926 12:38:22.319756 12033 heartbeater.cc:347] Connected to a master server at 127.10.224.190:46419
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:38:22.320102 12033 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:22.320676 12033 heartbeater.cc:511] Master 127.10.224.190:46419 requested a full tablet report, sending...
I20260926 12:38:22.321966 11708 ts_manager.cc:194] Registered new tserver with Master: fbc8217c23d7450fb498401da2826d23 (127.10.224.130:38027)
I20260926 12:38:22.322606 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:54801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:22.340036 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46419
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:22.455928 12038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:22.456292 12038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:22.456372 12038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:22.461232 12038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:22.461385 12038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:22.461493 12038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:22.467034 12038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:22.468304 12038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:22.469493 12038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:22.479597 12043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:22.481091 12038 server_base.cc:1034] running on GCE node
W20260926 12:38:22.481185 12046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:22.481590 12044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:22.482043 12038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:22.482755 12038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:22.483923 12038 hybrid_clock.cc:648] HybridClock initialized: now 1790426302483880 us; error 54 us; skew 500 ppm
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:22.486820 12038 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:22.487924 12038 webserver.cc:460] Webserver started at http://127.10.224.131:41635/ using document root <none> and password file <none>
I20260926 12:38:22.488514 12038 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:22.488575 12038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:22.488796 12038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:22.490945 12038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "ddada2aa6b144e07a9ed599cc487c74a"
format_stamp: "Formatted at 2026-09-26 12:38:22 on dist-test-slave-m3pj"
server_key: "01bc7d1bbaa47caa8dee2abaa9448b2f"
server_key_iv: "7eda519f11d5d9495052cc9659822183"
server_key_version: "encryptionkey@0"
I20260926 12:38:22.491617 12038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "ddada2aa6b144e07a9ed599cc487c74a"
format_stamp: "Formatted at 2026-09-26 12:38:22 on dist-test-slave-m3pj"
server_key: "01bc7d1bbaa47caa8dee2abaa9448b2f"
server_key_iv: "7eda519f11d5d9495052cc9659822183"
server_key_version: "encryptionkey@0"
I20260926 12:38:22.495636 12038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:22.498342 12053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:22.499797 12038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:22.499940 12038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "ddada2aa6b144e07a9ed599cc487c74a"
format_stamp: "Formatted at 2026-09-26 12:38:22 on dist-test-slave-m3pj"
server_key: "01bc7d1bbaa47caa8dee2abaa9448b2f"
server_key_iv: "7eda519f11d5d9495052cc9659822183"
server_key_version: "encryptionkey@0"
I20260926 12:38:22.500059 12038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:22.513650 12038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:22.517052 12038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:22.517268 12038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:22.517887 12038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:22.518741 12038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:22.518811 12038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:22.518890 12038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:22.518941 12038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:22.528155 12038 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:32781
I20260926 12:38:22.528175 12166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:32781 every 8 connection(s)
I20260926 12:38:22.529114 12038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:22.537106 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 12038
I20260926 12:38:22.537271 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:22.537546 11138 external_mini_cluster.cc:1428] Setting key 2b965731908e5680a7c40090836ea105
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:22.543685 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.531111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:38149 (local address 127.10.224.190:46419)
0926 12:38:22.531254 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.531258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.532184 (+ 926us) server_negotiation.cc:407] Connection header received
0926 12:38:22.533560 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.533563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.533621 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.533704 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:22.534915 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.535463 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.536714 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.536956 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.537073 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.540061 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:22.540064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:22.540066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:22.540091 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:22.541568 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.542115 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.542121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.542128 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.542201 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.542557 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.542562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.542564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.542756 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:22.542870 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.543376 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.543503 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:38:22.544591 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.531455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:46419 (local address 127.10.224.131:38149)
0926 12:38:22.532024 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:22.532070 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:22.533321 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:22.533887 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:22.533895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:22.534262 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:22.534756 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.534769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.535658 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.535661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.536584 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.536593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.537120 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.537124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.537346 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.538063 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:22.538080 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:22.539888 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:22.541694 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:22.541701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:22.541719 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:22.542003 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:22.542293 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:22.542297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:22.542299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:22.542460 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:22.542862 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:22.542869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:22.543139 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 12:38:22.543336 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 12:38:22.543733 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":133,"threads_started":1}
I20260926 12:38:22.545821 12167 heartbeater.cc:347] Connected to a master server at 127.10.224.190:46419
I20260926 12:38:22.546114 12167 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:22.546666 12167 heartbeater.cc:511] Master 127.10.224.190:46419 requested a full tablet report, sending...
I20260926 12:38:22.547847 11708 ts_manager.cc:194] Registered new tserver with Master: ddada2aa6b144e07a9ed599cc487c74a (127.10.224.131:32781)
I20260926 12:38:22.548436 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:38149
I20260926 12:38:22.551789 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:38:22.562849 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.555185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.10.224.190:46419)
0926 12:38:22.555337 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.555342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.555512 (+ 170us) server_negotiation.cc:407] Connection header received
0926 12:38:22.555679 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.555682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.555722 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.555790 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:22.556451 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.556964 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.557717 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.558013 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.558118 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.559591 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:22.559594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:22.559596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:22.559619 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:22.561100 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.561520 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.561523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.561525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.561630 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.561896 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.561899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.561900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.562133 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:22.562252 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.562495 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.562654 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:38:22.568398 11708 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:22.571915 11708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:22.587878 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.582440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.10.224.131:32781)
0926 12:38:22.583086 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.583099 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.583123 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:38:22.583199 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.583203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.583399 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.583629 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:22.584022 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.584875 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.586506 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.587113 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.587262 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.587426 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.587578 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.587683 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":130,"threads_started":1}
I20260926 12:38:22.587899 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.581550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:38027 (local address 127.0.0.1:52306)
0926 12:38:22.582571 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:22.582631 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:22.582769 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:22.583627 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:22.583630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:22.583649 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:22.583832 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.583842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.585384 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.585387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.586346 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.586355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.587426 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.587430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.587543 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.587570 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:38:22.587622 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 12:38:22.587710 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":24,"spinlock_wait_cycles":2944,"thread_start_us":97,"threads_started":1}
I20260926 12:38:22.587934 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.581757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:32781 (local address 127.0.0.1:57564)
0926 12:38:22.582617 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:22.582632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:22.582771 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:22.583597 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:22.583604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:22.583629 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:22.583886 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.583894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.585015 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.585022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.586376 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.586386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.587245 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.587249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.587362 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.587403 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 12:38:22.587558 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 12:38:22.587700 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":60,"spinlock_wait_cycles":4736,"thread_start_us":107,"threads_started":1}
I20260926 12:38:22.588511 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.581739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.10.224.130:38027)
0926 12:38:22.583088 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.583100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.583123 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:38:22.583202 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.583207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.583401 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.583983 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:22.584149 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.585236 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.586483 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.587283 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.588094 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.588219 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.588269 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.588357 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1236,"thread_start_us":130,"threads_started":1}
I20260926 12:38:22.589210 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.582001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:46635 (local address 127.0.0.1:41484)
0926 12:38:22.582862 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:22.582875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:22.582976 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:22.583780 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:22.583784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:22.583800 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:22.583944 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.583951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.585383 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.585387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.587584 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.587593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.588828 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.588831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.588935 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.588958 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:38:22.589026 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 12:38:22.589092 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":6,"thread_start_us":104,"threads_started":1}
I20260926 12:38:22.589777 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.582241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.10.224.129:46635)
0926 12:38:22.583206 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.583211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.583230 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:38:22.583292 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.583296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.583438 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.583656 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:22.584066 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.585247 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.587714 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.588496 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.588631 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.588762 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.589540 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.589645 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":102,"threads_started":1}
I20260926 12:38:22.591681 12101 tablet_service.cc:1467] Processing CreateTablet for tablet ee8c40b21251474c934937dd476cf191 (DEFAULT_TABLE table=test-table [id=efbc301d9aa345dbbf7148a95d3ca18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:22.591738 11967 tablet_service.cc:1467] Processing CreateTablet for tablet ee8c40b21251474c934937dd476cf191 (DEFAULT_TABLE table=test-table [id=efbc301d9aa345dbbf7148a95d3ca18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:22.592406 11833 tablet_service.cc:1467] Processing CreateTablet for tablet ee8c40b21251474c934937dd476cf191 (DEFAULT_TABLE table=test-table [id=efbc301d9aa345dbbf7148a95d3ca18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:22.592878 12101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8c40b21251474c934937dd476cf191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:22.592878 11967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8c40b21251474c934937dd476cf191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:22.593374 11833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8c40b21251474c934937dd476cf191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:22.599080 12184 tablet_bootstrap.cc:492] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a: Bootstrap starting.
I20260926 12:38:22.600674 12186 tablet_bootstrap.cc:492] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23: Bootstrap starting.
I20260926 12:38:22.602398 12184 tablet_bootstrap.cc:654] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:22.603349 12186 tablet_bootstrap.cc:654] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:22.603834 12185 tablet_bootstrap.cc:492] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95: Bootstrap starting.
I20260926 12:38:22.603826 12184 log.cc:824] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:22.604197 12186 log.cc:824] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:22.606020 12185 tablet_bootstrap.cc:654] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:22.606040 12184 tablet_bootstrap.cc:492] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a: No bootstrap required, opened a new log
I20260926 12:38:22.606251 12184 ts_tablet_manager.cc:1397] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:38:22.606863 12185 log.cc:824] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:22.606937 12186 tablet_bootstrap.cc:492] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23: No bootstrap required, opened a new log
I20260926 12:38:22.607118 12186 ts_tablet_manager.cc:1397] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:38:22.609114 12185 tablet_bootstrap.cc:492] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95: No bootstrap required, opened a new log
I20260926 12:38:22.609308 12185 ts_tablet_manager.cc:1397] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:38:22.609712 12184 raft_consensus.cc:348] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.609980 12184 raft_consensus.cc:374] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:22.610052 12184 raft_consensus.cc:729] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddada2aa6b144e07a9ed599cc487c74a, State: Initialized, Role: FOLLOWER
I20260926 12:38:22.610531 12184 consensus_queue.cc:260] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.611179 12167 heartbeater.cc:503] Master 127.10.224.190:46419 was elected leader, sending a full tablet report...
I20260926 12:38:22.611039 12186 raft_consensus.cc:348] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.611330 12186 raft_consensus.cc:374] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:22.611374 12186 raft_consensus.cc:729] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc8217c23d7450fb498401da2826d23, State: Initialized, Role: FOLLOWER
I20260926 12:38:22.611588 12184 ts_tablet_manager.cc:1428] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260926 12:38:22.611804 12186 consensus_queue.cc:260] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.612587 12033 heartbeater.cc:503] Master 127.10.224.190:46419 was elected leader, sending a full tablet report...
I20260926 12:38:22.612841 12186 ts_tablet_manager.cc:1428] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:38:22.613822 12185 raft_consensus.cc:348] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.614148 12185 raft_consensus.cc:374] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:22.614249 12185 raft_consensus.cc:729] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b30bb27623a4d09a89fd2548a733d95, State: Initialized, Role: FOLLOWER
I20260926 12:38:22.614746 12185 consensus_queue.cc:260] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.615810 11899 heartbeater.cc:503] Master 127.10.224.190:46419 was elected leader, sending a full tablet report...
I20260926 12:38:22.616819 12185 ts_tablet_manager.cc:1428] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 12:38:22.624228 12191 raft_consensus.cc:482] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:22.624425 12191 raft_consensus.cc:504] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.625586 12191 leader_election.cc:288] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddada2aa6b144e07a9ed599cc487c74a (127.10.224.131:32781), 1b30bb27623a4d09a89fd2548a733d95 (127.10.224.129:46635)
I20260926 12:38:22.629619 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.626132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:42705 (local address 127.10.224.129:46635)
0926 12:38:22.626308 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.626314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.626333 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:38:22.626394 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.626398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.626448 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.626527 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:22.626822 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.627408 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.628522 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.629013 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.629114 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.629154 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.629399 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.629482 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 12:38:22.629648 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.626011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:32841 (local address 127.10.224.131:32781)
0926 12:38:22.626288 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.626294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.626390 (+ 96us) server_negotiation.cc:407] Connection header received
0926 12:38:22.626565 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.626568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.626613 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.626692 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:22.626984 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.627449 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.628646 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.629106 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.629227 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.629276 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.629472 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.629544 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260926 12:38:22.629619 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.625900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:46635 (local address 127.10.224.130:42705)
0926 12:38:22.626095 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:22.626111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:22.626208 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:22.626555 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:22.626558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:22.626577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:22.626708 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.626714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.627545 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.627552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.628395 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.628404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.629149 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.629154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.629265 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.629291 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:38:22.629367 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 12:38:22.629467 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 12:38:22.629638 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.625924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:32781 (local address 127.10.224.130:32841)
0926 12:38:22.626280 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:22.626294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:22.626428 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:22.626713 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:22.626715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:22.626730 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:22.626876 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.626882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.627630 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.627633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.628499 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:22.628511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.629240 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.629243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:22.629319 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.629336 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:38:22.629440 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 12:38:22.629502 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260926 12:38:22.630371 12121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8c40b21251474c934937dd476cf191" candidate_uuid: "fbc8217c23d7450fb498401da2826d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddada2aa6b144e07a9ed599cc487c74a" is_pre_election: true
I20260926 12:38:22.630350 11853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8c40b21251474c934937dd476cf191" candidate_uuid: "fbc8217c23d7450fb498401da2826d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b30bb27623a4d09a89fd2548a733d95" is_pre_election: true
I20260926 12:38:22.630702 12121 raft_consensus.cc:2454] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc8217c23d7450fb498401da2826d23 in term 0.
I20260926 12:38:22.630702 11853 raft_consensus.cc:2454] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc8217c23d7450fb498401da2826d23 in term 0.
I20260926 12:38:22.631238 11921 leader_election.cc:302] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b30bb27623a4d09a89fd2548a733d95, ddada2aa6b144e07a9ed599cc487c74a, fbc8217c23d7450fb498401da2826d23; no voters:
I20260926 12:38:22.631628 12191 raft_consensus.cc:2781] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:22.631716 12191 raft_consensus.cc:482] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:22.631768 12191 raft_consensus.cc:3037] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:22.632813 12191 raft_consensus.cc:504] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.633225 12191 leader_election.cc:288] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [CANDIDATE]: Term 1 election: Requested vote from peers ddada2aa6b144e07a9ed599cc487c74a (127.10.224.131:32781), 1b30bb27623a4d09a89fd2548a733d95 (127.10.224.129:46635)
I20260926 12:38:22.633812 12121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8c40b21251474c934937dd476cf191" candidate_uuid: "fbc8217c23d7450fb498401da2826d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddada2aa6b144e07a9ed599cc487c74a"
I20260926 12:38:22.633931 12121 raft_consensus.cc:3037] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:22.634097 11853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8c40b21251474c934937dd476cf191" candidate_uuid: "fbc8217c23d7450fb498401da2826d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b30bb27623a4d09a89fd2548a733d95"
I20260926 12:38:22.634238 11853 raft_consensus.cc:3037] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:22.635129 12121 raft_consensus.cc:2454] T ee8c40b21251474c934937dd476cf191 P ddada2aa6b144e07a9ed599cc487c74a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc8217c23d7450fb498401da2826d23 in term 1.
I20260926 12:38:22.635387 11853 raft_consensus.cc:2454] T ee8c40b21251474c934937dd476cf191 P 1b30bb27623a4d09a89fd2548a733d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc8217c23d7450fb498401da2826d23 in term 1.
I20260926 12:38:22.635574 11921 leader_election.cc:302] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddada2aa6b144e07a9ed599cc487c74a, fbc8217c23d7450fb498401da2826d23; no voters:
I20260926 12:38:22.635773 12191 raft_consensus.cc:2781] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:22.636065 12191 raft_consensus.cc:686] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [term 1 LEADER]: Becoming Leader. State: Replica: fbc8217c23d7450fb498401da2826d23, State: Running, Role: LEADER
I20260926 12:38:22.636438 12191 consensus_queue.cc:237] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } }
I20260926 12:38:22.639947 11705 catalog_manager.cc:5495] T ee8c40b21251474c934937dd476cf191 P fbc8217c23d7450fb498401da2826d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc8217c23d7450fb498401da2826d23 (127.10.224.130). New cstate: current_term: 1 leader_uuid: "fbc8217c23d7450fb498401da2826d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8217c23d7450fb498401da2826d23" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddada2aa6b144e07a9ed599cc487c74a" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b30bb27623a4d09a89fd2548a733d95" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 46635 } health_report { overall_health: UNKNOWN } } }
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:22.658244 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.648602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.10.224.129:46635)
0926 12:38:22.648791 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.648795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.648867 (+ 72us) server_negotiation.cc:407] Connection header received
0926 12:38:22.648986 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.648989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.649030 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.649102 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:22.649768 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.650281 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.651304 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.651625 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.651710 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.654615 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:22.654628 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:22.654644 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:22.654673 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:22.656482 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.656918 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.656941 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.656942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.657023 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.657303 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.657305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.657306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.657578 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:22.657733 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.657899 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.658020 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:22.686554 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.677638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.10.224.129:46635)
0926 12:38:22.677806 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.677810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.677881 (+ 71us) server_negotiation.cc:407] Connection header received
0926 12:38:22.678002 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.678005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.678057 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.678128 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:22.678857 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.679352 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.680293 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.680544 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.680624 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.683339 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:22.683342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:22.683344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:22.683371 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:22.684926 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.685389 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.685391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.685392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.685495 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.685731 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.685733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.685735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.685889 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:22.685992 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.686247 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.686347 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:22.715054 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:22.705528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.10.224.129:46635)
0926 12:38:22.705693 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:22.705699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:22.705893 (+ 194us) server_negotiation.cc:407] Connection header received
0926 12:38:22.705979 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:22.705981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:22.706030 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:22.706109 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:22.706875 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.707692 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.708590 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:22.708966 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:22.709135 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:22.711823 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:22.711826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:22.711828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:22.711854 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:22.713309 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.713752 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.713758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.713760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.713861 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:22.714146 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:22.714151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:22.714152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:22.714343 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:22.714484 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:22.714731 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 12:38:22.714877 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 12:38:22.716619 11813 tablet_service.cc:2908] Rejecting scan request for tablet ee8c40b21251474c934937dd476cf191: Uninitialized: clean time has not yet been initialized
I20260926 12:38:22.718676 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 11770
I20260926 12:38:22.724771 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 11904
I20260926 12:38:22.730528 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 12038
I20260926 12:38:22.736320 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 11676
2026-09-26T12:38:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2363 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[12208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[12208](info): set up 2 sockets
Sep 26 12:38:22 dist-test-slave-m3pj krb5kdc[12208](info): commencing operation
krb5kdc: starting...
W20260926 12:38:23.799903 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:23 dist-test-slave-m3pj krb5kdc[12208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:23.927197 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:33947
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:33947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:24.040266 12221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:24.040606 12221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:24.040647 12221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:24.044677 12221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:24.044732 12221 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:38:24.044749 12221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:24.044768 12221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:24.044785 12221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:24.044800 12221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:24.050279 12221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:33947
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:33947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:24.051419 12221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:24.052577 12221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:24.061412 12227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:24.061401 12229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:24.061483 12226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:24.061673 12221 server_base.cc:1034] running on GCE node
I20260926 12:38:24.062292 12221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:24.063632 12221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:24.064811 12221 hybrid_clock.cc:648] HybridClock initialized: now 1790426304064770 us; error 64 us; skew 500 ppm
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:24.067907 12221 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:24.068967 12221 webserver.cc:460] Webserver started at http://127.10.224.190:44561/ using document root <none> and password file <none>
I20260926 12:38:24.069555 12221 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:24.069617 12221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:24.069850 12221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:24.072002 12221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "830cc44fe9ef455bb5d007feb4ee6f64"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "bbc0badd295aa95b616273534bf3d68b"
server_key_iv: "a4152d389ece7bfed8bf16a30efa1474"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.072592 12221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "830cc44fe9ef455bb5d007feb4ee6f64"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "bbc0badd295aa95b616273534bf3d68b"
server_key_iv: "a4152d389ece7bfed8bf16a30efa1474"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.076439 12221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:38:24.078819 12236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:24.080183 12221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:38:24.080303 12221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "830cc44fe9ef455bb5d007feb4ee6f64"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "bbc0badd295aa95b616273534bf3d68b"
server_key_iv: "a4152d389ece7bfed8bf16a30efa1474"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.080451 12221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:24.100248 12221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:24.104429 12221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:24.104671 12221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:24.114207 12221 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:33947
I20260926 12:38:24.114280 12298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:33947 every 8 connection(s)
I20260926 12:38:24.115278 12221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:24.118465 12299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:24.124369 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 12221
I20260926 12:38:24.124500 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:24.124778 11138 external_mini_cluster.cc:1428] Setting key 91ea90f7037083714b48597961d9fca1
I20260926 12:38:24.124754 12299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64: Bootstrap starting.
I20260926 12:38:24.127511 12299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:24.128403 12299 log.cc:824] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:24.130453 12299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64: No bootstrap required, opened a new log
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:24.134277 12299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830cc44fe9ef455bb5d007feb4ee6f64" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33947 } }
I20260926 12:38:24.134656 12299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:24.134748 12299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830cc44fe9ef455bb5d007feb4ee6f64, State: Initialized, Role: FOLLOWER
I20260926 12:38:24.135290 12299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830cc44fe9ef455bb5d007feb4ee6f64" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33947 } }
I20260926 12:38:24.135458 12299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:24.135579 12299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:24.135702 12299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:24.136648 12299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830cc44fe9ef455bb5d007feb4ee6f64" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33947 } }
I20260926 12:38:24.137018 12299 leader_election.cc:302] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 830cc44fe9ef455bb5d007feb4ee6f64; no voters:
I20260926 12:38:24.137360 12299 leader_election.cc:288] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:24.137511 12304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:24.137758 12304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [term 1 LEADER]: Becoming Leader. State: Replica: 830cc44fe9ef455bb5d007feb4ee6f64, State: Running, Role: LEADER
I20260926 12:38:24.138206 12304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830cc44fe9ef455bb5d007feb4ee6f64" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33947 } }
I20260926 12:38:24.138733 12299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:24.140348 12305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "830cc44fe9ef455bb5d007feb4ee6f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830cc44fe9ef455bb5d007feb4ee6f64" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33947 } } }
I20260926 12:38:24.140506 12305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:24.140482 12306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 830cc44fe9ef455bb5d007feb4ee6f64. Latest consensus state: current_term: 1 leader_uuid: "830cc44fe9ef455bb5d007feb4ee6f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830cc44fe9ef455bb5d007feb4ee6f64" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33947 } } }
I20260926 12:38:24.140609 12306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:24.141901 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.126404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.10.224.190:33947)
0926 12:38:24.126940 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.126956 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.126996 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:38:24.127877 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.127899 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.128271 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.128654 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:24.129488 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.130533 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.131419 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.132012 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.132209 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.135139 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:24.135145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:24.135161 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:24.135193 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:24.137631 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.138267 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.138273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.138281 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.138440 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.138792 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.138796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.138798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.139374 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:24.139601 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.140548 (+ 947us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.140962 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":162,"threads_started":1}
W20260926 12:38:24.144600 12319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:24.144668 12319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:24.144747 12318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:24.145511 12318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:24.151346 12318 catalog_manager.cc:1347] Generated new cluster ID: 338f73491962443ba6459038bf786dec
I20260926 12:38:24.151432 12318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:24.183269 12318 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:24.184561 12318 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:24.195062 12318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64: Generated new TSK 0
I20260926 12:38:24.195948 12318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:24.231978 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33947
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:24.355975 12325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:24.356369 12325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:24.356482 12325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:24.361361 12325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:24.361481 12325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:24.361557 12325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:38:24.361649 12325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:24.367954 12325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:33947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:24.369225 12325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:24.370347 12325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:24.380542 12330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:24.380605 12331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:24.380560 12333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:24.380854 12325 server_base.cc:1034] running on GCE node
I20260926 12:38:24.381325 12325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:24.381958 12325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:24.383157 12325 hybrid_clock.cc:648] HybridClock initialized: now 1790426304383134 us; error 34 us; skew 500 ppm
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:24.386183 12325 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:24.387507 12325 webserver.cc:460] Webserver started at http://127.10.224.129:46109/ using document root <none> and password file <none>
I20260926 12:38:24.388149 12325 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:24.388242 12325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:24.388468 12325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:24.390622 12325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "974e35cd505c49de93478c187740812e"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "16c46d78cf0099638165d5ee1e991404"
server_key_iv: "20e5985fd45dcda68c5895243e4b587b"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.391220 12325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "974e35cd505c49de93478c187740812e"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "16c46d78cf0099638165d5ee1e991404"
server_key_iv: "20e5985fd45dcda68c5895243e4b587b"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.395205 12325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 12:38:24.397951 12340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:24.399389 12325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 12:38:24.399575 12325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "974e35cd505c49de93478c187740812e"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "16c46d78cf0099638165d5ee1e991404"
server_key_iv: "20e5985fd45dcda68c5895243e4b587b"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.399708 12325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:24.416404 12325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:24.420871 12325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:24.421098 12325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:24.423273 12325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:24.423364 12325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:24.423431 12325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:24.423470 12325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:24.435833 12325 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:43031
I20260926 12:38:24.435855 12455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:43031 every 8 connection(s)
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:24.437021 12325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:24.439776 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 12325
I20260926 12:38:24.439894 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:24.440259 11138 external_mini_cluster.cc:1428] Setting key 3cee4752e52ab349ab4fffc434b33e2e
I20260926 12:38:24.444000 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.425475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:48647 (local address 127.10.224.190:33947)
0926 12:38:24.425685 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.425691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.427031 (+ 1340us) server_negotiation.cc:407] Connection header received
0926 12:38:24.428726 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.428734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.428807 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.428871 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:24.430940 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.431935 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.433774 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.434158 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.434365 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.437745 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:24.437750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:24.437752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:24.437781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:24.439809 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.440774 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.440777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.440779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.440900 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.441307 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.441312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.441313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.441642 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:24.441736 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.443681 (+ 1945us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.443852 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:38:24.444494 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.426316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33947 (local address 127.10.224.129:48647)
0926 12:38:24.426817 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.426871 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.428283 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.429314 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.429323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.430003 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:24.430701 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.430721 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.432087 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.432091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.433611 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.433624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.434584 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.434590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.434826 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.435656 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:24.435679 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:24.437543 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:24.439958 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.439967 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.439985 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.440411 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.441015 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.441018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.441020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.441189 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.441792 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:24.441800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:24.442129 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.442424 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.442886 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":138,"threads_started":1}
I20260926 12:38:24.446311 12456 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33947
I20260926 12:38:24.446646 12456 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:24.447296 12456 heartbeater.cc:511] Master 127.10.224.190:33947 requested a full tablet report, sending...
I20260926 12:38:24.449136 12253 ts_manager.cc:194] Registered new tserver with Master: 974e35cd505c49de93478c187740812e (127.10.224.129:43031)
I20260926 12:38:24.450482 12253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:48647
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:38:24.467968 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.450718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:52233 (local address 127.10.224.190:33947)
0926 12:38:24.450847 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.450853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.450867 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:38:24.450906 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.450909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.450951 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.451045 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:24.451616 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.452116 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.453364 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.453756 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.453835 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.463795 (+ 9960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:24.463802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:24.463806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:24.463849 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:24.465819 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.466396 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.466398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.466400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.466517 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.466869 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.466875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.466880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.467160 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:24.467295 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.467592 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.467811 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:38:24.468009 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.450365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33947 (local address 127.10.224.129:52233)
0926 12:38:24.450680 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.450695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.450784 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.451077 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.451079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.451306 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:24.451448 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.451455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.452244 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.452247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.453212 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.453223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.453999 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.454002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.454099 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.462964 (+ 8865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:24.462988 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:24.463589 (+ 601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:24.465959 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.465964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.465968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.466276 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.466650 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.466655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.466658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.466771 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.467287 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:24.467290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:24.467402 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.467520 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.467774 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:24.475042 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33947
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:24.588146 12465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:24.588500 12465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:24.588583 12465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:24.592691 12465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:24.592796 12465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:24.592878 12465 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:38:24.592967 12465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:24.598572 12465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:33947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:24.599846 12465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:24.601006 12465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:24.611558 12471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:24.611620 12473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:24.611855 12465 server_base.cc:1034] running on GCE node
W20260926 12:38:24.611635 12470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:24.612432 12465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:24.613121 12465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:24.614321 12465 hybrid_clock.cc:648] HybridClock initialized: now 1790426304614293 us; error 40 us; skew 500 ppm
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:24.617769 12465 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:24.618834 12465 webserver.cc:460] Webserver started at http://127.10.224.130:41027/ using document root <none> and password file <none>
I20260926 12:38:24.619465 12465 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:24.619598 12465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:24.619864 12465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:24.622051 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "573896c8d2ac49aea6680f73ef7ae6ab"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "7d6d4d0c0888a6f54ce165003cd658a3"
server_key_iv: "6339b3b654e0e9a3546110b517d67aeb"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.622676 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "573896c8d2ac49aea6680f73ef7ae6ab"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "7d6d4d0c0888a6f54ce165003cd658a3"
server_key_iv: "6339b3b654e0e9a3546110b517d67aeb"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.626578 12465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 12:38:24.629515 12480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:24.630918 12465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:38:24.631072 12465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "573896c8d2ac49aea6680f73ef7ae6ab"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "7d6d4d0c0888a6f54ce165003cd658a3"
server_key_iv: "6339b3b654e0e9a3546110b517d67aeb"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.631196 12465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:24.666015 12465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:24.669817 12465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:24.670033 12465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:24.671564 12465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:24.671648 12465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:24.671722 12465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:24.671769 12465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:24.682849 12465 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:37497
I20260926 12:38:24.682988 12595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:37497 every 8 connection(s)
I20260926 12:38:24.684091 12465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:24.685937 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.672618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:53865 (local address 127.10.224.190:33947)
0926 12:38:24.672772 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.672778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.673658 (+ 880us) server_negotiation.cc:407] Connection header received
0926 12:38:24.674772 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.674778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.674847 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.674950 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:24.676390 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.676909 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.678054 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.678279 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.678407 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.681738 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:24.681742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:24.681745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:24.681778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:24.683660 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.684313 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.684316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.684317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.684483 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.684919 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.684922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.684923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.685116 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:24.685196 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.685693 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.685796 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:38:24.686771 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.673028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33947 (local address 127.10.224.130:53865)
0926 12:38:24.673489 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.673539 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.674521 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.675180 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.675197 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.675587 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:24.676226 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.676242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.677038 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.677042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.677922 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.677931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.678457 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.678461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.678666 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.679484 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:24.679506 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:24.681538 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:24.683829 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.683835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.683849 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.684195 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.684605 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.684608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.684610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.684800 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.685217 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:24.685223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:24.685480 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.685638 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.685898 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":139,"threads_started":1}
I20260926 12:38:24.688308 12596 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33947
I20260926 12:38:24.688450 12596 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:24.688925 12596 heartbeater.cc:511] Master 127.10.224.190:33947 requested a full tablet report, sending...
I20260926 12:38:24.690097 12252 ts_manager.cc:194] Registered new tserver with Master: 573896c8d2ac49aea6680f73ef7ae6ab (127.10.224.130:37497)
I20260926 12:38:24.690824 12252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:53865
I20260926 12:38:24.692260 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 12465
I20260926 12:38:24.692377 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:24.692658 11138 external_mini_cluster.cc:1428] Setting key 5747672622a28cdf66cb4f2a16fc7289
I20260926 12:38:24.698422 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.690640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:44647 (local address 127.10.224.190:33947)
0926 12:38:24.690771 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.690776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.691009 (+ 233us) server_negotiation.cc:407] Connection header received
0926 12:38:24.691124 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.691127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.691177 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.691249 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:24.691770 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.692528 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.693522 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.693738 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.693868 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.695018 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:24.695025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:24.695031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:24.695064 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:24.696704 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.697171 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.697175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.697177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.697260 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.697554 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.697559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.697561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.697780 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:24.697888 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.698157 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.698268 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:38:24.698436 12602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.690573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33947 (local address 127.10.224.130:44647)
0926 12:38:24.690891 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.690907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.691006 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.691290 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.691293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.691497 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:24.691659 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.691665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.692669 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.692675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.693369 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.693379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.693860 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.693864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.693974 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.694553 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:24.694564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:24.694885 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:24.696842 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.696846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.696848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.697065 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.697352 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.697356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.697358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.697451 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.697888 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:24.697891 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:24.697988 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.698101 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.698258 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:24.723313 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33947
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:24.838722 12605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:24.839020 12605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:24.839068 12605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:24.843580 12605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:24.843659 12605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:24.843701 12605 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:38:24.843770 12605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:24.849534 12605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:33947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:24.850764 12605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:24.851887 12605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:24.862229 12611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:24.862279 12610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:24.862247 12613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:24.862304 12605 server_base.cc:1034] running on GCE node
I20260926 12:38:24.862941 12605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:24.863751 12605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:24.864959 12605 hybrid_clock.cc:648] HybridClock initialized: now 1790426304864928 us; error 42 us; skew 500 ppm
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:24.867976 12605 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:24.868965 12605 webserver.cc:460] Webserver started at http://127.10.224.131:36133/ using document root <none> and password file <none>
I20260926 12:38:24.869591 12605 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:24.869684 12605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:24.869926 12605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:24.872145 12605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "26afc3d44fc94245b21178eb0e68e7da"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "65c3f7b736b6d730b12edc0eb57505dc"
server_key_iv: "069855363ec0bb15b6e2427502098f3c"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.872767 12605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "26afc3d44fc94245b21178eb0e68e7da"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "65c3f7b736b6d730b12edc0eb57505dc"
server_key_iv: "069855363ec0bb15b6e2427502098f3c"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.876875 12605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:24.879691 12620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:24.881093 12605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:24.881258 12605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "26afc3d44fc94245b21178eb0e68e7da"
format_stamp: "Formatted at 2026-09-26 12:38:24 on dist-test-slave-m3pj"
server_key: "65c3f7b736b6d730b12edc0eb57505dc"
server_key_iv: "069855363ec0bb15b6e2427502098f3c"
server_key_version: "encryptionkey@0"
I20260926 12:38:24.881417 12605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:24.893940 12605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:24.897408 12605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:24.897619 12605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:24.899143 12605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:24.899222 12605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:24.899304 12605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:24.899351 12605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:24.910882 12605 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:38671
I20260926 12:38:24.911556 12735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:38671 every 8 connection(s)
I20260926 12:38:24.912248 12605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:24.913674 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.900133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:39845 (local address 127.10.224.190:33947)
0926 12:38:24.900308 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.900315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.901171 (+ 856us) server_negotiation.cc:407] Connection header received
0926 12:38:24.902372 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.902376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.902433 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.902519 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:24.904153 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.904785 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.906197 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.906428 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.906547 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.909371 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:24.909378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:24.909383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:24.909424 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:24.911271 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.911949 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.911953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.911955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.912052 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.912437 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.912440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.912442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.912710 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:24.912821 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.913447 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.913557 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:38:24.914551 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.900538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33947 (local address 127.10.224.131:39845)
0926 12:38:24.900996 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.901048 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.902072 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.902761 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.902771 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.903303 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:24.903997 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.904016 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.904919 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.904923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.906063 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.906073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.906601 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.906605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.906871 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.907638 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:24.907665 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:24.909181 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:24.911402 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.911408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.911425 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.911824 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.912150 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.912152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.912154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.912309 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.912875 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:24.912885 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:24.913186 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.913346 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.913609 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260926 12:38:24.915951 12736 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33947
I20260926 12:38:24.916088 12736 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:24.916697 12736 heartbeater.cc:511] Master 127.10.224.190:33947 requested a full tablet report, sending...
I20260926 12:38:24.917825 12252 ts_manager.cc:194] Registered new tserver with Master: 26afc3d44fc94245b21178eb0e68e7da (127.10.224.131:38671)
I20260926 12:38:24.918730 12252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:39845
I20260926 12:38:24.920157 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 12605
I20260926 12:38:24.920289 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:24.920588 11138 external_mini_cluster.cc:1428] Setting key 4fe9dd9d1c9cfd1a9b04f6249f5f2ff6
I20260926 12:38:24.922829 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:38:24.926542 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.918492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:36945 (local address 127.10.224.190:33947)
0926 12:38:24.918606 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.918609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.918819 (+ 210us) server_negotiation.cc:407] Connection header received
0926 12:38:24.918964 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.918967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.919012 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.919101 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:24.919637 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.920261 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.921401 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.921643 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.921762 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.923145 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:24.923151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:24.923160 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:24.923184 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:24.924749 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.925212 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.925217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.925220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.925313 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.925615 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.925621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.925623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.925824 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:24.925939 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.926232 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.926353 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:38:24.926555 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.918386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33947 (local address 127.10.224.131:36945)
0926 12:38:24.918733 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.918746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.918861 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.919115 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.919118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.919316 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:24.919469 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.919477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.920443 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.920447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.921258 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.921281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.921761 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.921765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.921866 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.922476 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:24.922488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:24.923001 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:24.924870 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.924878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.924880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.925103 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.925415 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:24.925420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:24.925422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:24.925517 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:24.925945 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:24.925951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:24.926054 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.926166 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.926340 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":140,"threads_started":1}
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:38:24 dist-test-slave-m3pj krb5kdc[12208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:24.952091 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.943027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.10.224.190:33947)
0926 12:38:24.943233 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.943236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.943251 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:24.943296 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.943298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.943341 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.943433 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:24.944239 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.944805 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.945652 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.945976 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.946079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.948874 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:24.948877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:24.948879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:24.948907 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:24.950367 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.950793 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.950795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.950796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.950875 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:24.951212 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:24.951216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:24.951217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:24.951410 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:24.951594 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.951860 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.951965 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 12:38:24.955476 12252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:24.958633 12252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:24.975155 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.968602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.10.224.130:37497)
0926 12:38:24.969755 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.969768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.969792 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:38:24.969896 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.969901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.970116 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.970289 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:24.970573 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.971831 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.973481 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.974373 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.974522 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.974664 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.974858 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.974976 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":124,"threads_started":1}
I20260926 12:38:24.975149 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.968235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:37497 (local address 127.0.0.1:50196)
0926 12:38:24.969153 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.969210 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.969363 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.970279 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.970285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.970317 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:24.970462 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.970469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.971971 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.971975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.973352 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.973362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.974503 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.974507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.974619 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.974662 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.974796 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.974954 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":75,"thread_start_us":113,"threads_started":1}
I20260926 12:38:24.975430 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.968501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38671 (local address 127.0.0.1:36412)
0926 12:38:24.969604 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.969618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.969781 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.970439 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.970442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.970473 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:24.970639 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.970646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.972015 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.972017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.973479 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.973489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.975048 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.975051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.975137 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.975160 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.975232 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.975317 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":379,"spinlock_wait_cycles":4608,"thread_start_us":101,"threads_started":1}
I20260926 12:38:24.975759 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.969469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.10.224.129:43031)
0926 12:38:24.969783 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.969790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.969929 (+ 139us) server_negotiation.cc:407] Connection header received
0926 12:38:24.970000 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.970005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.970149 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.970289 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:24.971676 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.972909 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.974066 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.974729 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.974849 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.974973 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.975224 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.975618 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
I20260926 12:38:24.975914 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.969008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.10.224.131:38671)
0926 12:38:24.969442 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:24.969457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:24.969710 (+ 253us) server_negotiation.cc:407] Connection header received
0926 12:38:24.969916 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:24.969923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:24.970127 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:24.970289 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:24.970762 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.971900 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.973622 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.974371 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.974521 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.974664 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:24.975677 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0926 12:38:24.975782 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":107,"threads_started":1}
I20260926 12:38:24.976019 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:24.969053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:43031 (local address 127.0.0.1:44106)
0926 12:38:24.969813 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:24.969828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:24.969918 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:24.970438 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:24.970442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:24.970472 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:24.970648 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.970654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.973034 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.973037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.973942 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:24.973950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:24.974854 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:24.974857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:24.974960 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:24.974983 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:38:24.975058 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 12:38:24.975127 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":20,"spinlock_wait_cycles":15360,"thread_start_us":79,"threads_started":1}
I20260926 12:38:24.978705 12530 tablet_service.cc:1467] Processing CreateTablet for tablet fc3465af5b0d4c2fa071f44a831ef500 (DEFAULT_TABLE table=test-table [id=bace55419a0d4b59a2e5f2bf5509e406]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:24.978791 12374 tablet_service.cc:1467] Processing CreateTablet for tablet fc3465af5b0d4c2fa071f44a831ef500 (DEFAULT_TABLE table=test-table [id=bace55419a0d4b59a2e5f2bf5509e406]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:24.978801 12670 tablet_service.cc:1467] Processing CreateTablet for tablet fc3465af5b0d4c2fa071f44a831ef500 (DEFAULT_TABLE table=test-table [id=bace55419a0d4b59a2e5f2bf5509e406]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:24.979944 12374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3465af5b0d4c2fa071f44a831ef500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:24.979945 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3465af5b0d4c2fa071f44a831ef500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:24.980244 12530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3465af5b0d4c2fa071f44a831ef500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:24.987869 12761 tablet_bootstrap.cc:492] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Bootstrap starting.
I20260926 12:38:24.988256 12760 tablet_bootstrap.cc:492] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: Bootstrap starting.
I20260926 12:38:24.990326 12760 tablet_bootstrap.cc:654] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:24.991252 12760 log.cc:824] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:24.991619 12759 tablet_bootstrap.cc:492] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Bootstrap starting.
I20260926 12:38:24.993487 12759 tablet_bootstrap.cc:654] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:24.994308 12759 log.cc:824] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:24.995905 12760 tablet_bootstrap.cc:492] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: No bootstrap required, opened a new log
I20260926 12:38:24.995904 12761 tablet_bootstrap.cc:654] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:24.996136 12760 ts_tablet_manager.cc:1397] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 12:38:24.996165 12759 tablet_bootstrap.cc:492] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: No bootstrap required, opened a new log
I20260926 12:38:24.996363 12759 ts_tablet_manager.cc:1397] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20260926 12:38:24.996778 12761 log.cc:824] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:24.998997 12761 tablet_bootstrap.cc:492] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: No bootstrap required, opened a new log
I20260926 12:38:24.999202 12761 ts_tablet_manager.cc:1397] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260926 12:38:25.000149 12759 raft_consensus.cc:348] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.000417 12759 raft_consensus.cc:374] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.000473 12759 raft_consensus.cc:729] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573896c8d2ac49aea6680f73ef7ae6ab, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.000738 12760 raft_consensus.cc:348] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.000927 12759 consensus_queue.cc:260] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.001026 12760 raft_consensus.cc:374] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.001080 12760 raft_consensus.cc:729] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974e35cd505c49de93478c187740812e, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.001639 12596 heartbeater.cc:503] Master 127.10.224.190:33947 was elected leader, sending a full tablet report...
I20260926 12:38:25.001590 12760 consensus_queue.cc:260] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.002050 12759 ts_tablet_manager.cc:1428] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 12:38:25.002621 12760 ts_tablet_manager.cc:1428] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 12:38:25.002655 12456 heartbeater.cc:503] Master 127.10.224.190:33947 was elected leader, sending a full tablet report...
I20260926 12:38:25.003221 12761 raft_consensus.cc:348] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.003464 12761 raft_consensus.cc:374] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.003509 12761 raft_consensus.cc:729] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26afc3d44fc94245b21178eb0e68e7da, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.003979 12761 consensus_queue.cc:260] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.004681 12736 heartbeater.cc:503] Master 127.10.224.190:33947 was elected leader, sending a full tablet report...
I20260926 12:38:25.004994 12761 ts_tablet_manager.cc:1428] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 12:38:25.124233 12766 raft_consensus.cc:482] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:25.124454 12766 raft_consensus.cc:504] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.125799 12766 leader_election.cc:288] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 573896c8d2ac49aea6680f73ef7ae6ab (127.10.224.130:37497), 26afc3d44fc94245b21178eb0e68e7da (127.10.224.131:38671)
I20260926 12:38:25.130306 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.126305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:45957 (local address 127.10.224.131:38671)
0926 12:38:25.126524 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.126531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.126781 (+ 250us) server_negotiation.cc:407] Connection header received
0926 12:38:25.126953 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.126959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.127020 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.127121 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:25.127446 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.128088 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.129343 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.129742 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.129890 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.129946 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.130138 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.130205 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 12:38:25.130307 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.126366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:42001 (local address 127.10.224.130:37497)
0926 12:38:25.126491 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.126496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.126852 (+ 356us) server_negotiation.cc:407] Connection header received
0926 12:38:25.127053 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.127056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.127103 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.127169 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:25.127484 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.128103 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.129280 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.129716 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.129848 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.129905 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.130089 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.130170 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:38:25.130332 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.126171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:37497 (local address 127.10.224.129:42001)
0926 12:38:25.126752 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.126769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.126887 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.127247 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.127250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.127263 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:25.127391 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.127400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.128252 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.128255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.129162 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.129171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.129847 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.129851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.129946 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.129978 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.130049 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.130144 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":304,"thread_start_us":111,"threads_started":1}
I20260926 12:38:25.130338 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.126171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38671 (local address 127.10.224.129:45957)
0926 12:38:25.126659 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.126673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.126800 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.127131 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.127134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.127150 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:25.127313 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.127328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.128215 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.128218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.129196 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.129206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.129893 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.129897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.129994 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.130012 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.130110 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.130176 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":48,"thread_start_us":121,"threads_started":1}
I20260926 12:38:25.131253 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3465af5b0d4c2fa071f44a831ef500" candidate_uuid: "974e35cd505c49de93478c187740812e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" is_pre_election: true
I20260926 12:38:25.131254 12690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3465af5b0d4c2fa071f44a831ef500" candidate_uuid: "974e35cd505c49de93478c187740812e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26afc3d44fc94245b21178eb0e68e7da" is_pre_election: true
I20260926 12:38:25.131616 12550 raft_consensus.cc:2454] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974e35cd505c49de93478c187740812e in term 0.
I20260926 12:38:25.131618 12690 raft_consensus.cc:2454] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974e35cd505c49de93478c187740812e in term 0.
I20260926 12:38:25.132140 12342 leader_election.cc:302] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573896c8d2ac49aea6680f73ef7ae6ab, 974e35cd505c49de93478c187740812e; no voters:
I20260926 12:38:25.132452 12766 raft_consensus.cc:2781] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:25.132525 12766 raft_consensus.cc:482] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:25.132555 12766 raft_consensus.cc:3037] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.133630 12766 raft_consensus.cc:504] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.134007 12766 leader_election.cc:288] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [CANDIDATE]: Term 1 election: Requested vote from peers 573896c8d2ac49aea6680f73ef7ae6ab (127.10.224.130:37497), 26afc3d44fc94245b21178eb0e68e7da (127.10.224.131:38671)
I20260926 12:38:25.134357 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3465af5b0d4c2fa071f44a831ef500" candidate_uuid: "974e35cd505c49de93478c187740812e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573896c8d2ac49aea6680f73ef7ae6ab"
I20260926 12:38:25.134377 12690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3465af5b0d4c2fa071f44a831ef500" candidate_uuid: "974e35cd505c49de93478c187740812e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26afc3d44fc94245b21178eb0e68e7da"
I20260926 12:38:25.134472 12550 raft_consensus.cc:3037] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.134507 12690 raft_consensus.cc:3037] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.135754 12550 raft_consensus.cc:2454] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974e35cd505c49de93478c187740812e in term 1.
I20260926 12:38:25.135913 12690 raft_consensus.cc:2454] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974e35cd505c49de93478c187740812e in term 1.
I20260926 12:38:25.136178 12342 leader_election.cc:302] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573896c8d2ac49aea6680f73ef7ae6ab, 974e35cd505c49de93478c187740812e; no voters:
I20260926 12:38:25.136359 12766 raft_consensus.cc:2781] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:25.136576 12766 raft_consensus.cc:686] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 1 LEADER]: Becoming Leader. State: Replica: 974e35cd505c49de93478c187740812e, State: Running, Role: LEADER
I20260926 12:38:25.136914 12766 consensus_queue.cc:237] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } }
I20260926 12:38:25.140617 12251 catalog_manager.cc:5495] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e reported cstate change: term changed from 0 to 1, leader changed from <none> to 974e35cd505c49de93478c187740812e (127.10.224.129). New cstate: current_term: 1 leader_uuid: "974e35cd505c49de93478c187740812e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } health_report { overall_health: HEALTHY } } }
W20260926 12:38:25.163718 12737 tablet.cc:2369] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:25.172492 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.168616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.10.224.129:43031)
0926 12:38:25.168778 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.168783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.168797 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:38:25.169046 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.169048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.169103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.169210 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:25.169491 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.169997 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.170893 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.171122 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.171234 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.171619 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:25.172046 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:25.172188 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.172268 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.172329 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:38:25.178696 12550 raft_consensus.cc:1261] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Refusing update from remote peer 974e35cd505c49de93478c187740812e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:25.178696 12690 raft_consensus.cc:1261] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Refusing update from remote peer 974e35cd505c49de93478c187740812e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:25.179396 12766 consensus_queue.cc:1035] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [LEADER]: Connected to new peer: Peer: permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:25.179607 12770 consensus_queue.cc:1035] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [LEADER]: Connected to new peer: Peer: permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:38:25.185490 12597 tablet.cc:2369] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:25.188519 12775 mvcc.cc:205] Tried to move back new op lower bound from 7333586146004381696 to 7333586145843650560. Current Snapshot: MvccSnapshot[applied={T|T < 7333586146004381696}]
I20260926 12:38:25.188946 12774 mvcc.cc:205] Tried to move back new op lower bound from 7333586146004381696 to 7333586145843650560. Current Snapshot: MvccSnapshot[applied={T|T < 7333586146004381696 or (T in {7333586146004381696})}]
I20260926 12:38:25.201804 12253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:25.202036 12253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:25.205065 12253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64: Sending DeleteTablet for 3 replicas of tablet fc3465af5b0d4c2fa071f44a831ef500
I20260926 12:38:25.206036 12530 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3465af5b0d4c2fa071f44a831ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:25 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:50196
I20260926 12:38:25.206041 12670 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3465af5b0d4c2fa071f44a831ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:25 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:36412
I20260926 12:38:25.206394 12374 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3465af5b0d4c2fa071f44a831ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:25 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:44106
I20260926 12:38:25.206610 12784 tablet_replica.cc:330] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: stopping tablet replica
I20260926 12:38:25.206864 12785 tablet_replica.cc:330] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: stopping tablet replica
I20260926 12:38:25.207094 12784 raft_consensus.cc:2229] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:25.207137 12785 raft_consensus.cc:2229] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:25.207437 12784 raft_consensus.cc:2258] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:25.207549 12785 raft_consensus.cc:2258] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:25.207499 12253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:25.207968 12253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:25.208961 12784 ts_tablet_manager.cc:1905] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:25.208961 12785 ts_tablet_manager.cc:1905] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:25.209688 12783 tablet_replica.cc:330] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: stopping tablet replica
I20260926 12:38:25.210044 12783 raft_consensus.cc:2229] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:25.210408 12783 raft_consensus.cc:2258] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:25.211969 12785 ts_tablet_manager.cc:1918] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:25.212076 12785 log.cc:1196] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/fc3465af5b0d4c2fa071f44a831ef500
I20260926 12:38:25.212093 12783 ts_tablet_manager.cc:1905] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:25.212417 12785 ts_tablet_manager.cc:1939] T fc3465af5b0d4c2fa071f44a831ef500 P 974e35cd505c49de93478c187740812e: Deleting consensus metadata
I20260926 12:38:25.213649 12240 catalog_manager.cc:4841] TS 974e35cd505c49de93478c187740812e (127.10.224.129:43031): tablet fc3465af5b0d4c2fa071f44a831ef500 (table test-table [id=bace55419a0d4b59a2e5f2bf5509e406]) successfully deleted
I20260926 12:38:25.214743 12374 tablet_service.cc:1467] Processing CreateTablet for tablet 13773cb019794f129792003075bccb81 (DEFAULT_TABLE table=test-table [id=da893d32f02e4987991e69ff3963b9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:25.215003 12374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13773cb019794f129792003075bccb81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:25.215005 12530 tablet_service.cc:1467] Processing CreateTablet for tablet 13773cb019794f129792003075bccb81 (DEFAULT_TABLE table=test-table [id=da893d32f02e4987991e69ff3963b9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:25.215243 12530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13773cb019794f129792003075bccb81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:25.216109 12784 ts_tablet_manager.cc:1918] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:25.216198 12784 log.cc:1196] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/fc3465af5b0d4c2fa071f44a831ef500
I20260926 12:38:25.216522 12784 ts_tablet_manager.cc:1939] T fc3465af5b0d4c2fa071f44a831ef500 P 573896c8d2ac49aea6680f73ef7ae6ab: Deleting consensus metadata
I20260926 12:38:25.217391 12760 tablet_bootstrap.cc:492] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: Bootstrap starting.
I20260926 12:38:25.217823 12238 catalog_manager.cc:4841] TS 573896c8d2ac49aea6680f73ef7ae6ab (127.10.224.130:37497): tablet fc3465af5b0d4c2fa071f44a831ef500 (table test-table [id=bace55419a0d4b59a2e5f2bf5509e406]) successfully deleted
I20260926 12:38:25.218325 12759 tablet_bootstrap.cc:492] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: Bootstrap starting.
I20260926 12:38:25.218384 12760 tablet_bootstrap.cc:654] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:25.219605 12759 tablet_bootstrap.cc:654] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:25.219641 12760 tablet_bootstrap.cc:492] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: No bootstrap required, opened a new log
I20260926 12:38:25.219713 12760 ts_tablet_manager.cc:1397] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 12:38:25.220253 12760 raft_consensus.cc:348] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.220402 12760 raft_consensus.cc:374] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.220448 12760 raft_consensus.cc:729] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974e35cd505c49de93478c187740812e, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.220563 12760 consensus_queue.cc:260] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.220847 12760 ts_tablet_manager.cc:1428] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:25.220849 12670 tablet_service.cc:1467] Processing CreateTablet for tablet 13773cb019794f129792003075bccb81 (DEFAULT_TABLE table=test-table [id=da893d32f02e4987991e69ff3963b9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:25.221117 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13773cb019794f129792003075bccb81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:25.222401 12783 ts_tablet_manager.cc:1918] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:25.222569 12783 log.cc:1196] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/fc3465af5b0d4c2fa071f44a831ef500
I20260926 12:38:25.222939 12783 ts_tablet_manager.cc:1939] T fc3465af5b0d4c2fa071f44a831ef500 P 26afc3d44fc94245b21178eb0e68e7da: Deleting consensus metadata
I20260926 12:38:25.223433 12761 tablet_bootstrap.cc:492] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: Bootstrap starting.
I20260926 12:38:25.224305 12240 catalog_manager.cc:4841] TS 26afc3d44fc94245b21178eb0e68e7da (127.10.224.131:38671): tablet fc3465af5b0d4c2fa071f44a831ef500 (table test-table [id=bace55419a0d4b59a2e5f2bf5509e406]) successfully deleted
I20260926 12:38:25.224421 12761 tablet_bootstrap.cc:654] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:25.225153 12759 tablet_bootstrap.cc:492] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: No bootstrap required, opened a new log
I20260926 12:38:25.225286 12759 ts_tablet_manager.cc:1397] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260926 12:38:25.225680 12761 tablet_bootstrap.cc:492] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: No bootstrap required, opened a new log
I20260926 12:38:25.225766 12761 ts_tablet_manager.cc:1397] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:38:25.225801 12759 raft_consensus.cc:348] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.225946 12759 raft_consensus.cc:374] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.226009 12759 raft_consensus.cc:729] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573896c8d2ac49aea6680f73ef7ae6ab, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.226162 12759 consensus_queue.cc:260] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.226219 12761 raft_consensus.cc:348] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.226316 12761 raft_consensus.cc:374] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.226361 12761 raft_consensus.cc:729] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26afc3d44fc94245b21178eb0e68e7da, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.226455 12759 ts_tablet_manager.cc:1428] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260926 12:38:25.226477 12761 consensus_queue.cc:260] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.226714 12761 ts_tablet_manager.cc:1428] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:25.382668 12767 raft_consensus.cc:482] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:25.382833 12767 raft_consensus.cc:504] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.384051 12767 leader_election.cc:288] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974e35cd505c49de93478c187740812e (127.10.224.129:43031), 573896c8d2ac49aea6680f73ef7ae6ab (127.10.224.130:37497)
I20260926 12:38:25.389819 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.384643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:57943 (local address 127.10.224.130:37497)
0926 12:38:25.384763 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.384767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.384783 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:38:25.384951 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.384955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.385041 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.385250 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:25.385645 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.386479 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.388348 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.389019 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.389158 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.389214 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.389586 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.389670 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:38:25.389819 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.384800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:54475 (local address 127.10.224.129:43031)
0926 12:38:25.384932 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.384937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.384960 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:38:25.385115 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.385118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.385171 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.385264 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:25.385645 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.386467 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.388348 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.389019 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.389163 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.389214 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.389586 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.389670 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:38:25.389889 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.384448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:37497 (local address 127.10.224.131:57943)
0926 12:38:25.384653 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.384670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.384804 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.385274 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.385277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.385294 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:25.385457 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.385467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.386595 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.386599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.388203 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.388216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.389160 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.389165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.389339 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.389458 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.389542 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.389709 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":250}
I20260926 12:38:25.389907 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.384448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:43031 (local address 127.10.224.131:54475)
0926 12:38:25.384879 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.384896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.384991 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.385310 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.385313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.385333 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:25.385508 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.385517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.386595 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.386599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.388203 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.388216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.389207 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.389211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.389343 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.389458 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.389542 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.389708 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":329,"thread_start_us":94,"threads_started":1}
I20260926 12:38:25.390476 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13773cb019794f129792003075bccb81" candidate_uuid: "26afc3d44fc94245b21178eb0e68e7da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" is_pre_election: true
I20260926 12:38:25.390573 12410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13773cb019794f129792003075bccb81" candidate_uuid: "26afc3d44fc94245b21178eb0e68e7da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974e35cd505c49de93478c187740812e" is_pre_election: true
I20260926 12:38:25.390655 12550 raft_consensus.cc:2454] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26afc3d44fc94245b21178eb0e68e7da in term 0.
I20260926 12:38:25.390805 12410 raft_consensus.cc:2454] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26afc3d44fc94245b21178eb0e68e7da in term 0.
I20260926 12:38:25.391135 12622 leader_election.cc:302] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26afc3d44fc94245b21178eb0e68e7da, 573896c8d2ac49aea6680f73ef7ae6ab; no voters:
I20260926 12:38:25.391611 12767 raft_consensus.cc:2781] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:25.391670 12767 raft_consensus.cc:482] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:25.391700 12767 raft_consensus.cc:3037] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.392827 12767 raft_consensus.cc:504] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.393185 12767 leader_election.cc:288] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [CANDIDATE]: Term 1 election: Requested vote from peers 974e35cd505c49de93478c187740812e (127.10.224.129:43031), 573896c8d2ac49aea6680f73ef7ae6ab (127.10.224.130:37497)
I20260926 12:38:25.393627 12410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13773cb019794f129792003075bccb81" candidate_uuid: "26afc3d44fc94245b21178eb0e68e7da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974e35cd505c49de93478c187740812e"
I20260926 12:38:25.393627 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13773cb019794f129792003075bccb81" candidate_uuid: "26afc3d44fc94245b21178eb0e68e7da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573896c8d2ac49aea6680f73ef7ae6ab"
I20260926 12:38:25.393772 12550 raft_consensus.cc:3037] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.393772 12410 raft_consensus.cc:3037] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.395045 12550 raft_consensus.cc:2454] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26afc3d44fc94245b21178eb0e68e7da in term 1.
I20260926 12:38:25.395045 12410 raft_consensus.cc:2454] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26afc3d44fc94245b21178eb0e68e7da in term 1.
I20260926 12:38:25.395403 12624 leader_election.cc:302] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26afc3d44fc94245b21178eb0e68e7da, 974e35cd505c49de93478c187740812e; no voters:
I20260926 12:38:25.395627 12767 raft_consensus.cc:2781] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:25.395861 12767 raft_consensus.cc:686] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 1 LEADER]: Becoming Leader. State: Replica: 26afc3d44fc94245b21178eb0e68e7da, State: Running, Role: LEADER
I20260926 12:38:25.396138 12767 consensus_queue.cc:237] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.399144 12251 catalog_manager.cc:5495] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da reported cstate change: term changed from 0 to 1, leader changed from <none> to 26afc3d44fc94245b21178eb0e68e7da (127.10.224.131). New cstate: current_term: 1 leader_uuid: "26afc3d44fc94245b21178eb0e68e7da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } health_report { overall_health: HEALTHY } } }
W20260926 12:38:25.439265 12457 tablet.cc:2369] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 12:38:25 dist-test-slave-m3pj krb5kdc[12208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:25.455217 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.445608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.10.224.190:33947)
0926 12:38:25.445760 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.445764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.446008 (+ 244us) server_negotiation.cc:407] Connection header received
0926 12:38:25.446134 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.446136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.446178 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.446249 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:25.447075 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.447769 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.448914 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.449134 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.449215 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.451832 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:25.451836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:25.451839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:25.451864 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:25.453425 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:25.453956 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:25.453959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:25.453960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:25.454042 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:25.454306 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:25.454308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:25.454309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:25.454533 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:25.454612 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.454940 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.455085 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":53}
I20260926 12:38:25.455289 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.445566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33947 (local address 127.0.0.1:47610)
0926 12:38:25.445903 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.445920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.446042 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.446293 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.446296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.446839 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:25.446952 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.446960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.447881 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.447885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.448792 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.448802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.449247 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.449250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.449342 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.449958 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:25.449973 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:25.451677 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:25.453562 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:25.453571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:25.453578 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:25.453842 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:25.454142 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:25.454144 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:25.454146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:25.454228 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:25.454655 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:25.454662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:25.454773 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.454888 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.455025 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":140,"threads_started":1}
I20260926 12:38:25.459813 12251 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:47610:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:38:25.460714 12251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:25.465950 12374 tablet_service.cc:1467] Processing CreateTablet for tablet 0be39e713bb7480ab661d2d03a056c72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c27ee7fd5a145da9dfed54904aa34e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:38:25.466013 12530 tablet_service.cc:1467] Processing CreateTablet for tablet 0be39e713bb7480ab661d2d03a056c72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c27ee7fd5a145da9dfed54904aa34e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:38:25.466126 12670 tablet_service.cc:1467] Processing CreateTablet for tablet 0be39e713bb7480ab661d2d03a056c72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c27ee7fd5a145da9dfed54904aa34e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:38:25.466408 12374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be39e713bb7480ab661d2d03a056c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:25.466416 12530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be39e713bb7480ab661d2d03a056c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:25.466476 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be39e713bb7480ab661d2d03a056c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:25.469223 12760 tablet_bootstrap.cc:492] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: Bootstrap starting.
I20260926 12:38:25.469542 12759 tablet_bootstrap.cc:492] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: Bootstrap starting.
I20260926 12:38:25.470221 12760 tablet_bootstrap.cc:654] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:25.470332 12761 tablet_bootstrap.cc:492] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: Bootstrap starting.
I20260926 12:38:25.470556 12759 tablet_bootstrap.cc:654] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:25.471122 12761 tablet_bootstrap.cc:654] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:25.471886 12759 tablet_bootstrap.cc:492] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: No bootstrap required, opened a new log
I20260926 12:38:25.471993 12759 ts_tablet_manager.cc:1397] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 12:38:25.472277 12760 tablet_bootstrap.cc:492] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: No bootstrap required, opened a new log
I20260926 12:38:25.472308 12761 tablet_bootstrap.cc:492] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: No bootstrap required, opened a new log
I20260926 12:38:25.472378 12760 ts_tablet_manager.cc:1397] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:38:25.472394 12761 ts_tablet_manager.cc:1397] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:38:25.472450 12759 raft_consensus.cc:348] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.472577 12759 raft_consensus.cc:374] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.472615 12759 raft_consensus.cc:729] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573896c8d2ac49aea6680f73ef7ae6ab, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.472745 12759 consensus_queue.cc:260] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.472826 12761 raft_consensus.cc:348] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.472849 12760 raft_consensus.cc:348] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.472939 12761 raft_consensus.cc:374] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.472977 12760 raft_consensus.cc:374] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:25.472977 12761 raft_consensus.cc:729] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26afc3d44fc94245b21178eb0e68e7da, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.473019 12760 raft_consensus.cc:729] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974e35cd505c49de93478c187740812e, State: Initialized, Role: FOLLOWER
I20260926 12:38:25.473102 12761 consensus_queue.cc:260] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.473156 12760 consensus_queue.cc:260] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.473302 12759 ts_tablet_manager.cc:1428] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:25.473264 12765 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.473371 12761 ts_tablet_manager.cc:1428] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:25.473397 12765 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:25.473408 12760 ts_tablet_manager.cc:1428] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:25.473533 12767 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.473668 12767 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:25.473616 12778 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.473742 12778 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:25.555403 12765 raft_consensus.cc:482] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:25.555634 12765 raft_consensus.cc:504] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.556886 12765 leader_election.cc:288] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974e35cd505c49de93478c187740812e (127.10.224.129:43031), 26afc3d44fc94245b21178eb0e68e7da (127.10.224.131:38671)
I20260926 12:38:25.561214 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.557260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:47305 (local address 127.10.224.129:43031)
0926 12:38:25.557448 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.557455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.557542 (+ 87us) server_negotiation.cc:407] Connection header received
0926 12:38:25.557721 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.557725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.557777 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.557886 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:25.558247 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.558888 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.560165 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.560624 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.560719 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.560760 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.560999 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.561078 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:38:25.561239 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.557118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:43031 (local address 127.10.224.130:47305)
0926 12:38:25.557440 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.557456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.557606 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.557904 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.557910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.557935 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:25.558109 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.558115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.559002 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.559006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.560037 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.560046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.560730 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.560734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.560841 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.560869 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.560956 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.561038 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":60,"spinlock_wait_cycles":2432,"thread_start_us":88,"threads_started":1}
I20260926 12:38:25.561491 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.557583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38671 (local address 127.10.224.130:52871)
0926 12:38:25.557876 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.557890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.557997 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.558281 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.558284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.558301 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:25.558474 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.558481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.559257 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.559259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.560219 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.560229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.561041 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.561044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.561137 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.561159 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.561239 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.561335 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":4,"spinlock_wait_cycles":896,"thread_start_us":69,"threads_started":1}
I20260926 12:38:25.561499 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.557702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:52871 (local address 127.10.224.131:38671)
0926 12:38:25.557884 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.557888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.557941 (+ 53us) server_negotiation.cc:407] Connection header received
0926 12:38:25.558116 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.558119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.558177 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.558262 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:25.558583 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.559141 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.560353 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.560924 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.561032 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.561076 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.561289 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.561374 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:38:25.561715 12410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be39e713bb7480ab661d2d03a056c72" candidate_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974e35cd505c49de93478c187740812e" is_pre_election: true
I20260926 12:38:25.561854 12410 raft_consensus.cc:2454] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573896c8d2ac49aea6680f73ef7ae6ab in term 0.
I20260926 12:38:25.562238 12484 leader_election.cc:302] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573896c8d2ac49aea6680f73ef7ae6ab, 974e35cd505c49de93478c187740812e; no voters:
I20260926 12:38:25.562400 12690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be39e713bb7480ab661d2d03a056c72" candidate_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26afc3d44fc94245b21178eb0e68e7da" is_pre_election: true
I20260926 12:38:25.562570 12690 raft_consensus.cc:2454] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573896c8d2ac49aea6680f73ef7ae6ab in term 0.
I20260926 12:38:25.562590 12765 raft_consensus.cc:2781] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:25.562644 12765 raft_consensus.cc:482] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:25.562716 12765 raft_consensus.cc:3037] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.563896 12765 raft_consensus.cc:504] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.564230 12765 leader_election.cc:288] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [CANDIDATE]: Term 1 election: Requested vote from peers 974e35cd505c49de93478c187740812e (127.10.224.129:43031), 26afc3d44fc94245b21178eb0e68e7da (127.10.224.131:38671)
I20260926 12:38:25.564606 12690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be39e713bb7480ab661d2d03a056c72" candidate_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26afc3d44fc94245b21178eb0e68e7da"
I20260926 12:38:25.564602 12410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be39e713bb7480ab661d2d03a056c72" candidate_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974e35cd505c49de93478c187740812e"
I20260926 12:38:25.564713 12410 raft_consensus.cc:3037] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.564742 12690 raft_consensus.cc:3037] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:25.565661 12690 raft_consensus.cc:2454] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573896c8d2ac49aea6680f73ef7ae6ab in term 1.
I20260926 12:38:25.565661 12410 raft_consensus.cc:2454] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573896c8d2ac49aea6680f73ef7ae6ab in term 1.
I20260926 12:38:25.565964 12484 leader_election.cc:302] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26afc3d44fc94245b21178eb0e68e7da, 573896c8d2ac49aea6680f73ef7ae6ab; no voters:
I20260926 12:38:25.566124 12765 raft_consensus.cc:2781] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:25.566337 12765 raft_consensus.cc:686] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 LEADER]: Becoming Leader. State: Replica: 573896c8d2ac49aea6680f73ef7ae6ab, State: Running, Role: LEADER
I20260926 12:38:25.566643 12765 consensus_queue.cc:237] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } }
I20260926 12:38:25.567971 12803 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: TxnStatusTablet state changed. Reason: New leader 573896c8d2ac49aea6680f73ef7ae6ab. Latest consensus state: current_term: 1 leader_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.568092 12803 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:38:25.568410 12805 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:38:25.568574 12805 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:38:25.569309 12251 catalog_manager.cc:5495] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 573896c8d2ac49aea6680f73ef7ae6ab (127.10.224.130). New cstate: current_term: 1 leader_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } health_report { overall_health: UNKNOWN } } }
I20260926 12:38:25.600733 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.596466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.10.224.130:37497)
0926 12:38:25.596611 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.596617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.596634 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:38:25.596831 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.596834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.596913 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.597022 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:25.597302 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.598081 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.598965 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.599308 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.599412 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.599805 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:25.600265 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:25.600413 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.600505 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.600561 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:38:25.600708 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.596355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:37497 (local address 127.0.0.1:50198)
0926 12:38:25.596552 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.596574 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.596669 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.597067 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.597070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.597077 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:25.597187 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.597192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.598189 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.598191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.598845 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.598854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.599430 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.599433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.599503 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.599622 (+ 119us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:25.600395 (+ 773us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:25.600411 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.600483 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.600578 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 12:38:25.603966 12690 raft_consensus.cc:1261] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Refusing update from remote peer 573896c8d2ac49aea6680f73ef7ae6ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:25.604027 12410 raft_consensus.cc:1261] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Refusing update from remote peer 573896c8d2ac49aea6680f73ef7ae6ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:25.604554 12803 consensus_queue.cc:1035] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:25.604801 12765 consensus_queue.cc:1035] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:25.606211 12767 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: TxnStatusTablet state changed. Reason: New leader 573896c8d2ac49aea6680f73ef7ae6ab. Latest consensus state: current_term: 1 leader_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.606338 12767 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:25.606946 12778 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: TxnStatusTablet state changed. Reason: New leader 573896c8d2ac49aea6680f73ef7ae6ab. Latest consensus state: current_term: 1 leader_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.607064 12778 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:25.609441 12765 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.609583 12765 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:38:25.609498 12803 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.610486 12773 mvcc.cc:205] Tried to move back new op lower bound from 7333586147747725312 to 7333586147603591168. Current Snapshot: MvccSnapshot[applied={T|T < 7333586147747725312}]
I20260926 12:38:25.610652 12778 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.610764 12778 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 974e35cd505c49de93478c187740812e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:25.610771 12767 tablet_replica.cc:439] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 } } peers { permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 } } peers { permanent_uuid: "26afc3d44fc94245b21178eb0e68e7da" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38671 } } }
I20260926 12:38:25.610880 12767 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 26afc3d44fc94245b21178eb0e68e7da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:25.611079 12803 tablet_replica.cc:442] T 0be39e713bb7480ab661d2d03a056c72 P 573896c8d2ac49aea6680f73ef7ae6ab: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:38:25.620042 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.615702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.10.224.131:38671)
0926 12:38:25.616043 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.616047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.616057 (+ 10us) server_negotiation.cc:407] Connection header received
0926 12:38:25.616124 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.616126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.616177 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.616300 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:25.616571 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.617138 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.618323 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.618570 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.618664 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.618976 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:25.619610 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:25.619713 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.619811 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.619873 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260926 12:38:25.632000 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.628138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:55245 (local address 127.10.224.130:37497)
0926 12:38:25.628280 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.628285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.628517 (+ 232us) server_negotiation.cc:407] Connection header received
0926 12:38:25.628611 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.628613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.628658 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.628728 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:25.629006 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.629503 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.630760 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.631102 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.631191 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.631573 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:25.631653 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:25.631709 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.631827 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.631890 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:38:25.632093 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.628061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:37497 (local address 127.10.224.131:55245)
0926 12:38:25.628401 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.628417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.628527 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.628756 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.628759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.628767 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:25.628906 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.628912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.629612 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.629615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.630624 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.630635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.631242 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.631246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.631340 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.631395 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:25.631734 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:25.631741 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.631811 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.631931 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":127,"threads_started":1}
I20260926 12:38:25.639358 12550 raft_consensus.cc:1261] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Refusing update from remote peer 26afc3d44fc94245b21178eb0e68e7da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:25.639599 12410 raft_consensus.cc:1261] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Refusing update from remote peer 26afc3d44fc94245b21178eb0e68e7da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:25.639837 12790 consensus_queue.cc:1035] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [LEADER]: Connected to new peer: Peer: permanent_uuid: "573896c8d2ac49aea6680f73ef7ae6ab" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 37497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:25.640053 12767 consensus_queue.cc:1035] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [LEADER]: Connected to new peer: Peer: permanent_uuid: "974e35cd505c49de93478c187740812e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:25.663045 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.658699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:41087 (local address 127.10.224.131:38671)
0926 12:38:25.658824 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.658828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.659021 (+ 193us) server_negotiation.cc:407] Connection header received
0926 12:38:25.659131 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.659134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.659173 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.659232 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:25.659555 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.660190 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.661967 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.662188 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.662273 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.662613 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:25.662688 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:25.662746 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.662874 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.662925 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:38:25.663115 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.658568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38671 (local address 127.10.224.130:41087)
0926 12:38:25.658931 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:25.658947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:25.659053 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:25.659268 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:25.659271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:25.659277 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:25.659411 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.659418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.660308 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.660312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.661835 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:25.661847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.662305 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.662308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:25.662406 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.662473 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:25.662781 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:25.662788 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:38:25.662856 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 12:38:25.662945 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":113,"threads_started":1}
I20260926 12:38:25.688827 12253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:25.689057 12253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:25.691751 12253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 830cc44fe9ef455bb5d007feb4ee6f64: Sending DeleteTablet for 3 replicas of tablet 13773cb019794f129792003075bccb81
I20260926 12:38:25.692503 12374 tablet_service.cc:1514] Processing DeleteTablet for tablet 13773cb019794f129792003075bccb81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:25 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:44106
I20260926 12:38:25.692714 12785 tablet_replica.cc:330] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: stopping tablet replica
I20260926 12:38:25.692835 12785 raft_consensus.cc:2229] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:25.692953 12785 raft_consensus.cc:2258] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:25.693882 12670 tablet_service.cc:1514] Processing DeleteTablet for tablet 13773cb019794f129792003075bccb81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:25 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:36412
I20260926 12:38:25.694060 12783 tablet_replica.cc:330] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: stopping tablet replica
I20260926 12:38:25.694165 12783 raft_consensus.cc:2229] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:25.694478 12783 raft_consensus.cc:2258] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:25.695497 12785 ts_tablet_manager.cc:1905] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:25.698705 12785 ts_tablet_manager.cc:1918] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:25.698786 12785 log.cc:1196] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/13773cb019794f129792003075bccb81
I20260926 12:38:25.699041 12785 ts_tablet_manager.cc:1939] T 13773cb019794f129792003075bccb81 P 974e35cd505c49de93478c187740812e: Deleting consensus metadata
I20260926 12:38:25.699496 12530 tablet_service.cc:1514] Processing DeleteTablet for tablet 13773cb019794f129792003075bccb81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:25 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:50196
I20260926 12:38:25.699783 12784 tablet_replica.cc:330] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: stopping tablet replica
I20260926 12:38:25.699895 12784 raft_consensus.cc:2229] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:25.700011 12784 raft_consensus.cc:2258] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:25.700047 12783 ts_tablet_manager.cc:1905] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:25.700343 12240 catalog_manager.cc:4841] TS 974e35cd505c49de93478c187740812e (127.10.224.129:43031): tablet 13773cb019794f129792003075bccb81 (table test-table [id=da893d32f02e4987991e69ff3963b9d7]) successfully deleted
I20260926 12:38:25.702847 12783 ts_tablet_manager.cc:1918] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:25.702932 12783 log.cc:1196] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/13773cb019794f129792003075bccb81
I20260926 12:38:25.703164 12783 ts_tablet_manager.cc:1939] T 13773cb019794f129792003075bccb81 P 26afc3d44fc94245b21178eb0e68e7da: Deleting consensus metadata
I20260926 12:38:25.704180 12784 ts_tablet_manager.cc:1905] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:25.706879 12784 ts_tablet_manager.cc:1918] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:25.706967 12784 log.cc:1196] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/13773cb019794f129792003075bccb81
I20260926 12:38:25.707221 12784 ts_tablet_manager.cc:1939] T 13773cb019794f129792003075bccb81 P 573896c8d2ac49aea6680f73ef7ae6ab: Deleting consensus metadata
I20260926 12:38:25.708019 12238 catalog_manager.cc:4841] TS 573896c8d2ac49aea6680f73ef7ae6ab (127.10.224.130:37497): tablet 13773cb019794f129792003075bccb81 (table test-table [id=da893d32f02e4987991e69ff3963b9d7]) successfully deleted
I20260926 12:38:25.708751 12240 catalog_manager.cc:4841] TS 26afc3d44fc94245b21178eb0e68e7da (127.10.224.131:38671): tablet 13773cb019794f129792003075bccb81 (table test-table [id=da893d32f02e4987991e69ff3963b9d7]) successfully deleted
Sep 26 12:38:25 dist-test-slave-m3pj krb5kdc[12208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:25.722940 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.708030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.10.224.129:43031)
0926 12:38:25.708162 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.708167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.709345 (+ 1178us) server_negotiation.cc:407] Connection header received
0926 12:38:25.709579 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.709582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.709638 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.709693 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:25.710732 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.711509 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.712748 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.713077 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.713162 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.717775 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:25.717793 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:25.717812 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:25.717853 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:25.720320 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:25.720923 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:25.720931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:25.720933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:25.721047 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:25.721412 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:25.721426 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:25.721428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:25.721790 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:25.722010 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.722591 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.722759 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 12:38:25.724506 12450 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44110
I20260926 12:38:25.736316 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:25.726530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.10.224.190:33947)
0926 12:38:25.726668 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:25.726675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:25.727127 (+ 452us) server_negotiation.cc:407] Connection header received
0926 12:38:25.727231 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:25.727234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:25.727292 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:25.727394 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:25.728214 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.729039 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.730212 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:25.730557 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:25.730675 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:25.732447 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:25.732451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:25.732454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:25.732484 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:25.734284 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:25.734833 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:25.734837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:25.734839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:25.734933 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:25.735317 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:25.735321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:25.735323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:25.735627 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:25.735759 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:25.736024 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 12:38:25.736146 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 12:38:25.737188 12253 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47626
I20260926 12:38:25.738737 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 12325
I20260926 12:38:25.749593 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 12465
I20260926 12:38:25.757347 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 12605
I20260926 12:38:25.763931 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 12221
2026-09-26T12:38:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3026 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:25 dist-test-slave-m3pj krb5kdc[12827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:25 dist-test-slave-m3pj krb5kdc[12827](info): set up 2 sockets
Sep 26 12:38:25 dist-test-slave-m3pj krb5kdc[12827](info): commencing operation
krb5kdc: starting...
W20260926 12:38:26.803802 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:26 dist-test-slave-m3pj krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:26.934919 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:42819
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:42819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:27.050532 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:27.050899 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:27.050983 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:27.055130 12840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:27.055241 12840 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:38:27.055274 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:27.055322 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:27.055387 12840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:27.055428 12840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:27.060997 12840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:42819
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:42819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:27.062270 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:27.063418 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:27.072407 12845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:27.072492 12846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:27.072414 12848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:27.072655 12840 server_base.cc:1034] running on GCE node
I20260926 12:38:27.073487 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:27.074647 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:27.075825 12840 hybrid_clock.cc:648] HybridClock initialized: now 1790426307075797 us; error 44 us; skew 500 ppm
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:27.079300 12840 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:27.080289 12840 webserver.cc:460] Webserver started at http://127.10.224.190:38355/ using document root <none> and password file <none>
I20260926 12:38:27.080839 12840 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:27.080912 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:27.081086 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:27.083348 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "7591d95c7d374de5b4529ff7fd57afda"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "94848239eb5eeb59b106197d70039130"
server_key_iv: "2a5549f16496f5a576484ee4ed4c5592"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.084023 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "7591d95c7d374de5b4529ff7fd57afda"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "94848239eb5eeb59b106197d70039130"
server_key_iv: "2a5549f16496f5a576484ee4ed4c5592"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.087968 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:27.090993 12855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:27.092635 12840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:27.092792 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "7591d95c7d374de5b4529ff7fd57afda"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "94848239eb5eeb59b106197d70039130"
server_key_iv: "2a5549f16496f5a576484ee4ed4c5592"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.092928 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:27.110564 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:27.114514 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:27.114753 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:27.123759 12840 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:42819
I20260926 12:38:27.123822 12917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:42819 every 8 connection(s)
I20260926 12:38:27.124927 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:27.128237 12918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:27.132509 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 12840
I20260926 12:38:27.132622 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:27.132944 11138 external_mini_cluster.cc:1428] Setting key beaea813c174c1739b2c33575a29bb1a
I20260926 12:38:27.135067 12918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: Bootstrap starting.
I20260926 12:38:27.137580 12918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:27.138407 12918 log.cc:824] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:27.140568 12918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: No bootstrap required, opened a new log
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:27.145018 12918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7591d95c7d374de5b4529ff7fd57afda" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42819 } }
I20260926 12:38:27.145332 12918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:27.145430 12918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7591d95c7d374de5b4529ff7fd57afda, State: Initialized, Role: FOLLOWER
I20260926 12:38:27.145946 12918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7591d95c7d374de5b4529ff7fd57afda" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42819 } }
I20260926 12:38:27.146119 12918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:27.146221 12918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:27.146354 12918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:27.147416 12918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7591d95c7d374de5b4529ff7fd57afda" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42819 } }
I20260926 12:38:27.147858 12918 leader_election.cc:302] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7591d95c7d374de5b4529ff7fd57afda; no voters:
I20260926 12:38:27.148308 12918 leader_election.cc:288] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:27.148445 12923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:27.148694 12923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [term 1 LEADER]: Becoming Leader. State: Replica: 7591d95c7d374de5b4529ff7fd57afda, State: Running, Role: LEADER
I20260926 12:38:27.149063 12923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7591d95c7d374de5b4529ff7fd57afda" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42819 } }
I20260926 12:38:27.149628 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.134616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.10.224.190:42819)
0926 12:38:27.135116 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:27.135132 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:27.135170 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:38:27.136013 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:27.136035 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:27.136409 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:27.136784 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:27.137672 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.138916 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.139959 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.140482 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.140676 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.143458 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:27.143465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:27.143482 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:27.143563 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:27.145748 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.146340 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.146349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.146357 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.146555 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.146926 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.146930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.146932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.147426 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:27.147711 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:27.148024 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 12:38:27.148440 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":153,"threads_started":1}
I20260926 12:38:27.149830 12918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:27.150430 12923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7591d95c7d374de5b4529ff7fd57afda. Latest consensus state: current_term: 1 leader_uuid: "7591d95c7d374de5b4529ff7fd57afda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7591d95c7d374de5b4529ff7fd57afda" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42819 } } }
I20260926 12:38:27.150542 12923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:27.153120 12924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7591d95c7d374de5b4529ff7fd57afda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7591d95c7d374de5b4529ff7fd57afda" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42819 } } }
I20260926 12:38:27.153218 12924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda [sys.catalog]: This master's current role is: LEADER
W20260926 12:38:27.155298 12939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:27.155396 12939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:27.155491 12927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:27.156453 12927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:27.162501 12927 catalog_manager.cc:1347] Generated new cluster ID: 0c0e71de54d74bfca40e89db5e4bc6fb
I20260926 12:38:27.162606 12927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:27.172353 12927 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:27.173491 12927 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:27.178681 12927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: Generated new TSK 0
I20260926 12:38:27.179539 12927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:27.214833 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42819
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:27.362182 12944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:27.362484 12944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:27.362538 12944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:27.366884 12944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:27.366986 12944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:27.367045 12944 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:38:27.367065 12944 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:38:27.367120 12944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:27.372727 12944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:42819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:27.373878 12944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:27.374891 12944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:27.384996 12952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:27.385142 12949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:27.384996 12950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:27.385318 12944 server_base.cc:1034] running on GCE node
I20260926 12:38:27.385810 12944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:27.386533 12944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:27.387832 12944 hybrid_clock.cc:648] HybridClock initialized: now 1790426307387810 us; error 68 us; skew 500 ppm
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:27.390779 12944 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:27.391868 12944 webserver.cc:460] Webserver started at http://127.10.224.129:39177/ using document root <none> and password file <none>
I20260926 12:38:27.392481 12944 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:27.392601 12944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:27.392853 12944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:27.395095 12944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "1034bdda96f24a99a0a95649b7632c2f"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "58e8a3d3f38eb3e54d5f26246acc1d6d"
server_key_iv: "6754c33c5b26a9462612874606e5a0b1"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.395771 12944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "1034bdda96f24a99a0a95649b7632c2f"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "58e8a3d3f38eb3e54d5f26246acc1d6d"
server_key_iv: "6754c33c5b26a9462612874606e5a0b1"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.400269 12944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:38:27.403084 12959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:27.404558 12944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:27.404704 12944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "1034bdda96f24a99a0a95649b7632c2f"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "58e8a3d3f38eb3e54d5f26246acc1d6d"
server_key_iv: "6754c33c5b26a9462612874606e5a0b1"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.404834 12944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:27.425273 12944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:27.428992 12944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:27.429243 12944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:27.430892 12944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:27.430998 12944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:27.431069 12944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:27.431120 12944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:27.443465 12944 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:40987
I20260926 12:38:27.443496 13074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:40987 every 8 connection(s)
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:27.444746 12944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:27.449237 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.432225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:35479 (local address 127.10.224.190:42819)
0926 12:38:27.432394 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:27.432399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:27.433595 (+ 1196us) server_negotiation.cc:407] Connection header received
0926 12:38:27.434965 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:27.434975 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:27.435044 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:27.435160 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:27.437286 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.438258 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.439941 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.440349 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.440533 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.444502 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:27.444508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:27.444511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:27.444550 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:27.446593 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.447260 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.447263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.447265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.447353 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.447829 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.447832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.447833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.448081 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:27.448173 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:27.448955 (+ 782us) server_negotiation.cc:299] Negotiation successful
0926 12:38:27.449100 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:38:27.450492 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.432818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42819 (local address 127.10.224.129:35479)
0926 12:38:27.433379 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:27.433438 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:27.434624 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:27.435388 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:27.435398 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:27.436122 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:27.437030 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.437074 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.438498 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.438502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.439752 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.439768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.440635 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.440639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.440851 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.442135 (+ 1284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:27.442171 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:27.444224 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:27.446760 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.446767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.446782 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.447131 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.447442 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.447444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.447446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.447688 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.448220 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:27.448227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:27.448641 (+ 414us) client_negotiation.cc:769] Sending connection context
0926 12:38:27.448851 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 12:38:27.449220 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260926 12:38:27.452669 13075 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42819
I20260926 12:38:27.452958 13075 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:27.453428 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 12944
I20260926 12:38:27.453562 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:27.453732 13075 heartbeater.cc:511] Master 127.10.224.190:42819 requested a full tablet report, sending...
I20260926 12:38:27.453897 11138 external_mini_cluster.cc:1428] Setting key 72c289f9d9a499cf67750c0e40e63747
I20260926 12:38:27.455755 12872 ts_manager.cc:194] Registered new tserver with Master: 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987)
I20260926 12:38:27.457227 12872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:35479
I20260926 12:38:27.463939 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.455659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:49257 (local address 127.10.224.190:42819)
0926 12:38:27.455818 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:27.455823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:27.456249 (+ 426us) server_negotiation.cc:407] Connection header received
0926 12:38:27.456365 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:27.456368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:27.456415 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:27.456501 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:27.457135 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.457852 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.458884 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.459123 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.459238 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.460691 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:27.460693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:27.460696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:27.460718 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:27.462227 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.462674 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.462678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.462680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.462759 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.463046 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.463049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.463051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.463234 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:27.463344 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:27.463625 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 12:38:27.463739 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":61}
I20260926 12:38:27.463951 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.455769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42819 (local address 127.10.224.129:49257)
0926 12:38:27.456139 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:27.456155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:27.456278 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:27.456528 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:27.456534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:27.456804 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:27.456988 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.457000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.458017 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.458021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.458755 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.458765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.459245 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.459249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.459366 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.459995 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:27.460015 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:27.460538 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:27.462353 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.462359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.462361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.462570 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.462853 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.462857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.462859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.462948 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.463337 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:27.463340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:27.463441 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 12:38:27.463572 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 12:38:27.463733 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:27.483246 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42819
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:27.608982 13084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:27.609392 13084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:27.609519 13084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:27.614363 13084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:27.614491 13084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:27.614581 13084 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:38:27.614632 13084 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:38:27.614701 13084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:27.621119 13084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:42819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:27.622455 13084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:27.623651 13084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:27.634865 13092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:27.634860 13090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:27.634858 13089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:27.635187 13084 server_base.cc:1034] running on GCE node
I20260926 12:38:27.635891 13084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:27.636659 13084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:27.637905 13084 hybrid_clock.cc:648] HybridClock initialized: now 1790426307637887 us; error 39 us; skew 500 ppm
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:27.641389 13084 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:27.642462 13084 webserver.cc:460] Webserver started at http://127.10.224.130:32957/ using document root <none> and password file <none>
I20260926 12:38:27.643126 13084 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:27.643231 13084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:27.643512 13084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:27.646025 13084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "d55f9c919de44a9783165a90d2215c3f"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "ae370c010b123f6584a0b99d234e2412"
server_key_iv: "96018943d3d5365e89b692789cedf798"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.646711 13084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "d55f9c919de44a9783165a90d2215c3f"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "ae370c010b123f6584a0b99d234e2412"
server_key_iv: "96018943d3d5365e89b692789cedf798"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.651619 13084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 12:38:27.654927 13099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:27.656463 13084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:38:27.656634 13084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "d55f9c919de44a9783165a90d2215c3f"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "ae370c010b123f6584a0b99d234e2412"
server_key_iv: "96018943d3d5365e89b692789cedf798"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.656773 13084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:27.672081 13084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:27.676038 13084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:27.676285 13084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:27.678016 13084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:27.678102 13084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:27.678184 13084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:27.678232 13084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:27.690929 13084 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:38403
I20260926 12:38:27.690987 13214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:38403 every 8 connection(s)
I20260926 12:38:27.692153 13084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:27.694304 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.679072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:55031 (local address 127.10.224.190:42819)
0926 12:38:27.679233 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:27.679239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:27.680230 (+ 991us) server_negotiation.cc:407] Connection header received
0926 12:38:27.681395 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:27.681399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:27.681454 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:27.681512 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:27.683509 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.684541 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.685862 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.686101 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.686222 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.689801 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:27.689805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:27.689810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:27.689843 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:27.691648 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.692391 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.692395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.692396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.692484 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.692907 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.692911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.692913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.693157 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:27.693238 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:27.693974 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 12:38:27.694119 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:38:27.695148 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.679592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42819 (local address 127.10.224.130:55031)
0926 12:38:27.680057 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:27.680108 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:27.681111 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:27.682049 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:27.682059 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:27.682636 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:27.683312 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.683334 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.684696 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.684699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.685718 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.685727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.686276 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.686280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.686552 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.687265 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:27.687286 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:27.689597 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:27.691778 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.691785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.691801 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.692185 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.692591 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.692594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.692597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.692776 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.693285 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:27.693292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:27.693672 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 12:38:27.693924 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 12:38:27.694275 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":117,"threads_started":1}
I20260926 12:38:27.696406 13215 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42819
I20260926 12:38:27.696593 13215 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:27.697162 13215 heartbeater.cc:511] Master 127.10.224.190:42819 requested a full tablet report, sending...
I20260926 12:38:27.698372 12871 ts_manager.cc:194] Registered new tserver with Master: d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403)
I20260926 12:38:27.699241 12871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:55031
I20260926 12:38:27.700484 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 13084
I20260926 12:38:27.700613 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:27.700899 11138 external_mini_cluster.cc:1428] Setting key 841d262b2138154fae8a93b709640e38
I20260926 12:38:27.707692 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.698912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42819 (local address 127.10.224.130:34775)
0926 12:38:27.699238 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:27.699256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:27.699380 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:27.699680 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:27.699684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:27.699921 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:27.700039 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.700045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.701113 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.701118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.701956 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.701965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.702547 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.702551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.702666 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.703195 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:27.703218 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:27.703897 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:27.705945 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.705947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.705949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.706161 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.706462 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.706465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.706466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.706547 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.707138 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:27.707141 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:27.707224 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 12:38:27.707321 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 12:38:27.707440 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
I20260926 12:38:27.707860 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.699038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:34775 (local address 127.10.224.190:42819)
0926 12:38:27.699164 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:27.699168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:27.699362 (+ 194us) server_negotiation.cc:407] Connection header received
0926 12:38:27.699467 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:27.699470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:27.699563 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:27.699646 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:27.700158 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.700987 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.702136 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.702420 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.702508 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.704034 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:27.704037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:27.704039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:27.704061 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:27.705821 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.706262 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.706268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.706273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.706364 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.706652 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.706658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.706662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.706999 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:27.707116 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:27.707412 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 12:38:27.707608 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:27.731845 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42819
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:27.850005 13224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:27.850301 13224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:27.850351 13224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:27.854746 13224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:27.854828 13224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:27.854873 13224 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:38:27.854888 13224 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:38:27.854954 13224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:27.860673 13224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:42819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:27.861938 13224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:27.863056 13224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:27.873278 13230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:27.873258 13229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:27.873332 13224 server_base.cc:1034] running on GCE node
W20260926 12:38:27.873252 13232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:27.873929 13224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:27.874665 13224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:27.875862 13224 hybrid_clock.cc:648] HybridClock initialized: now 1790426307875843 us; error 45 us; skew 500 ppm
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:27.878865 13224 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:27.880095 13224 webserver.cc:460] Webserver started at http://127.10.224.131:35997/ using document root <none> and password file <none>
I20260926 12:38:27.880748 13224 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:27.880852 13224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:27.881116 13224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:27.883411 13224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "b1c17095522c4defb17da62f3338b0b8"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "3d025bf981c4f912beb8070ba0ddb7b7"
server_key_iv: "d4afb3ebeff4e86886d6c910d48d78df"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.884101 13224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "b1c17095522c4defb17da62f3338b0b8"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "3d025bf981c4f912beb8070ba0ddb7b7"
server_key_iv: "d4afb3ebeff4e86886d6c910d48d78df"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.888425 13224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:27.891444 13239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:27.893028 13224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:27.893190 13224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "b1c17095522c4defb17da62f3338b0b8"
format_stamp: "Formatted at 2026-09-26 12:38:27 on dist-test-slave-m3pj"
server_key: "3d025bf981c4f912beb8070ba0ddb7b7"
server_key_iv: "d4afb3ebeff4e86886d6c910d48d78df"
server_key_version: "encryptionkey@0"
I20260926 12:38:27.893390 13224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:27.943231 13224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:27.946730 13224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:27.946949 13224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:27.948552 13224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:27.948635 13224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:27.948735 13224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:27.948779 13224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:27.960081 13224 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:39963
I20260926 12:38:27.960150 13354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:39963 every 8 connection(s)
I20260926 12:38:27.961171 13224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:27.963924 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.949639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:58319 (local address 127.10.224.190:42819)
0926 12:38:27.949775 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:27.949779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:27.951070 (+ 1291us) server_negotiation.cc:407] Connection header received
0926 12:38:27.952574 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:27.952578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:27.952637 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:27.952723 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:27.954446 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.955369 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.956737 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.957065 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.957186 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.959910 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:27.959914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:27.959916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:27.959957 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:27.961732 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.962436 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.962439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.962440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.962566 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.962870 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.962872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.962873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.963052 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:27.963124 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:27.963697 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 12:38:27.963801 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:38:27.965018 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.950298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42819 (local address 127.10.224.131:58319)
0926 12:38:27.950870 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:27.950935 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:27.952353 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:27.952917 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:27.952928 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:27.953446 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:27.954213 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.954235 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.955549 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.955555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.956600 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.956609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.957225 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.957230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.957418 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.958084 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:27.958108 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:27.959713 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:27.961855 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.961861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.961877 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.962213 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.962648 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.962650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.962652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.962801 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.963139 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:27.963145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:27.963394 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 12:38:27.963646 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 12:38:27.963956 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":127,"threads_started":1}
I20260926 12:38:27.966530 13355 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42819
I20260926 12:38:27.966732 13355 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:27.967401 13355 heartbeater.cc:511] Master 127.10.224.190:42819 requested a full tablet report, sending...
I20260926 12:38:27.968641 12872 ts_manager.cc:194] Registered new tserver with Master: b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963)
I20260926 12:38:27.969404 12872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:58319
I20260926 12:38:27.969769 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 13224
I20260926 12:38:27.969900 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:27.970185 11138 external_mini_cluster.cc:1428] Setting key 172871d3abeed33894922d218af79d9d
I20260926 12:38:27.972333 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:38:27.977002 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.968590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42819 (local address 127.10.224.131:58227)
0926 12:38:27.968891 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:27.968904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:27.968991 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:27.969211 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:27.969213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:27.969381 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:27.969513 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.969522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.970452 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.970457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.971251 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:27.971260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.971889 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.971893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:27.972114 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.972776 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:27.972795 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:27.973284 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:27.975319 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.975322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.975324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.975609 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.975984 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:27.975986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:27.975987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:27.976062 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:27.976558 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:27.976560 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:27.976642 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 12:38:27.976722 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:38:27.976839 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":16,"thread_start_us":111,"threads_started":1}
I20260926 12:38:27.977239 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.968685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:58227 (local address 127.10.224.190:42819)
0926 12:38:27.968802 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:27.968807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:27.968976 (+ 169us) server_negotiation.cc:407] Connection header received
0926 12:38:27.969066 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:27.969068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:27.969111 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:27.969205 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:27.969636 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.970331 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.971426 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.971764 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.971872 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.973424 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:27.973426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:27.973428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:27.973452 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:27.975200 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.975735 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.975739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.975741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.975871 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:27.976150 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:27.976157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:27.976162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:27.976457 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:27.976576 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:27.976822 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 12:38:27.976977 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:38:27 dist-test-slave-m3pj krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:28.002110 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:27.992936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.10.224.190:42819)
0926 12:38:27.993085 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:27.993089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:27.993104 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:27.993150 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:27.993153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:27.993196 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:27.993300 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:27.993997 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.994579 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.995406 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:27.995698 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:27.995786 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:27.998758 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:27.998762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:27.998765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:27.998795 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:28.000356 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:28.000822 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:28.000825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:28.000829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:28.000914 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:28.001200 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:28.001203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:28.001205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:28.001419 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:28.001559 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.001810 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.001930 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:38:28.005298 12872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:28.008858 12872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:28.025575 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.019720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:38403 (local address 127.0.0.1:53720)
0926 12:38:28.020155 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.020170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.020264 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.020989 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.020995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.021021 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.021117 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.021123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.022667 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.022674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.023841 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.023851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.024964 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.024969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.025090 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.025142 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.025242 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.025392 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":89,"thread_start_us":157,"threads_started":1}
I20260926 12:38:28.025873 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.018898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:39963 (local address 127.0.0.1:35254)
0926 12:38:28.019372 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.019425 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.019608 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.020740 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.020746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.020771 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.020941 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.020954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.022581 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.022587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.024286 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.024297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.025466 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.025470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.025560 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.025586 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.025643 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.025726 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":122,"thread_start_us":90,"threads_started":1}
I20260926 12:38:28.026149 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.018922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40987 (local address 127.0.0.1:60346)
0926 12:38:28.020021 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.020038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.020137 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.021143 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.021145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.021160 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.021304 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.021310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.022750 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.022753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.024242 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.024255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.025644 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.025647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.025792 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.025823 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.025902 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.025985 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":26,"thread_start_us":126,"threads_started":1}
I20260926 12:38:28.026336 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.020220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.10.224.130:38403)
0926 12:38:28.020510 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.020522 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.020545 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:38:28.020618 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.020622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.020826 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.020998 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.021239 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.022479 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.023986 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.024807 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.025858 (+ 1051us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.026014 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.026076 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.026179 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":122,"threads_started":1}
I20260926 12:38:28.026531 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.019735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.10.224.131:39963)
0926 12:38:28.020290 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.020302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.020322 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:38:28.020388 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.020393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.020568 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.020749 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.021092 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.022446 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.024433 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.025327 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.026086 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.026208 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.026257 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.026369 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":136,"threads_started":1}
I20260926 12:38:28.026716 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.019796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.10.224.129:40987)
0926 12:38:28.020645 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.020651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.020670 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:38:28.020731 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.020736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.020878 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.020998 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.021457 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.022639 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.024455 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.025267 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.025421 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.025587 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.026464 (+ 877us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.026585 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":107,"threads_started":1}
I20260926 12:38:28.029568 13149 tablet_service.cc:1467] Processing CreateTablet for tablet de8c7308d96047dba37e588a5bdbdc6d (DEFAULT_TABLE table=test-table [id=ec427583c8754b0e8ca3af95155bcf91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:28.029551 13009 tablet_service.cc:1467] Processing CreateTablet for tablet de8c7308d96047dba37e588a5bdbdc6d (DEFAULT_TABLE table=test-table [id=ec427583c8754b0e8ca3af95155bcf91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:28.029563 13289 tablet_service.cc:1467] Processing CreateTablet for tablet de8c7308d96047dba37e588a5bdbdc6d (DEFAULT_TABLE table=test-table [id=ec427583c8754b0e8ca3af95155bcf91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:28.030593 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8c7308d96047dba37e588a5bdbdc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.030658 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8c7308d96047dba37e588a5bdbdc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.030659 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8c7308d96047dba37e588a5bdbdc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.038146 13378 tablet_bootstrap.cc:492] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Bootstrap starting.
I20260926 12:38:28.039569 13379 tablet_bootstrap.cc:492] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Bootstrap starting.
I20260926 12:38:28.039887 13380 tablet_bootstrap.cc:492] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Bootstrap starting.
I20260926 12:38:28.041406 13378 tablet_bootstrap.cc:654] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.041930 13380 tablet_bootstrap.cc:654] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.041931 13379 tablet_bootstrap.cc:654] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.042294 13378 log.cc:824] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:28.042840 13379 log.cc:824] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:28.042840 13380 log.cc:824] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:28.044659 13379 tablet_bootstrap.cc:492] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: No bootstrap required, opened a new log
I20260926 12:38:28.044654 13378 tablet_bootstrap.cc:492] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: No bootstrap required, opened a new log
I20260926 12:38:28.044866 13380 tablet_bootstrap.cc:492] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: No bootstrap required, opened a new log
I20260926 12:38:28.044925 13378 ts_tablet_manager.cc:1397] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:38:28.044903 13379 ts_tablet_manager.cc:1397] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:38:28.045038 13380 ts_tablet_manager.cc:1397] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:38:28.049451 13379 raft_consensus.cc:348] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.049754 13379 raft_consensus.cc:374] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.049819 13379 raft_consensus.cc:729] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1034bdda96f24a99a0a95649b7632c2f, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.049697 13378 raft_consensus.cc:348] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.049697 13380 raft_consensus.cc:348] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.049985 13380 raft_consensus.cc:374] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.049985 13378 raft_consensus.cc:374] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.050050 13380 raft_consensus.cc:729] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c17095522c4defb17da62f3338b0b8, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.050050 13378 raft_consensus.cc:729] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55f9c919de44a9783165a90d2215c3f, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.050486 13379 consensus_queue.cc:260] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.050565 13380 consensus_queue.cc:260] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.050565 13378 consensus_queue.cc:260] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.051225 13075 heartbeater.cc:503] Master 127.10.224.190:42819 was elected leader, sending a full tablet report...
I20260926 12:38:28.051728 13380 ts_tablet_manager.cc:1428] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 12:38:28.051728 13378 ts_tablet_manager.cc:1428] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:38:28.051752 13379 ts_tablet_manager.cc:1428] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 12:38:28.052112 13215 heartbeater.cc:503] Master 127.10.224.190:42819 was elected leader, sending a full tablet report...
I20260926 12:38:28.052138 13355 heartbeater.cc:503] Master 127.10.224.190:42819 was elected leader, sending a full tablet report...
I20260926 12:38:28.137116 13386 raft_consensus.cc:482] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:28.137321 13386 raft_consensus.cc:504] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.138517 13386 leader_election.cc:288] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987), d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403)
I20260926 12:38:28.142840 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.138818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40987 (local address 127.10.224.131:43171)
0926 12:38:28.139034 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.139051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.139157 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.139443 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.139446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.139467 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.139612 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.139619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.140469 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.140472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.141521 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.141529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.142287 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.142291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.142469 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.142496 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.142577 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.142657 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 12:38:28.142899 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.138983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:43171 (local address 127.10.224.129:40987)
0926 12:38:28.139140 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.139145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.139164 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:38:28.139254 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.139257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.139310 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.139402 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.139757 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.140353 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.141675 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.142154 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.142266 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.142322 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.142640 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.142757 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:38:28.143673 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8c7308d96047dba37e588a5bdbdc6d" candidate_uuid: "b1c17095522c4defb17da62f3338b0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1034bdda96f24a99a0a95649b7632c2f" is_pre_election: true
I20260926 12:38:28.144073 13029 raft_consensus.cc:2454] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1c17095522c4defb17da62f3338b0b8 in term 0.
I20260926 12:38:28.144199 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.138827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:38403 (local address 127.10.224.131:60925)
0926 12:38:28.139710 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.139726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.139901 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.140188 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.140191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.140208 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.140872 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.140882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.141753 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.141755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.142670 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.142678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.143752 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.143755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.143853 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.143876 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.143960 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.144030 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":354,"thread_start_us":189,"threads_started":1}
I20260926 12:38:28.144284 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.139048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:60925 (local address 127.10.224.130:38403)
0926 12:38:28.139167 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.139171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.139797 (+ 626us) server_negotiation.cc:407] Connection header received
0926 12:38:28.140043 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.140046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.140089 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.140178 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.141064 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.141630 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.142831 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.143571 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.143709 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.143753 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.144022 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.144130 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:38:28.144646 13243 leader_election.cc:302] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1034bdda96f24a99a0a95649b7632c2f, b1c17095522c4defb17da62f3338b0b8; no voters:
I20260926 12:38:28.144970 13386 raft_consensus.cc:2781] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:28.145082 13386 raft_consensus.cc:482] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:28.145129 13386 raft_consensus.cc:3037] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.145061 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8c7308d96047dba37e588a5bdbdc6d" candidate_uuid: "b1c17095522c4defb17da62f3338b0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55f9c919de44a9783165a90d2215c3f" is_pre_election: true
I20260926 12:38:28.145337 13169 raft_consensus.cc:2454] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1c17095522c4defb17da62f3338b0b8 in term 0.
I20260926 12:38:28.146437 13386 raft_consensus.cc:504] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.146782 13386 leader_election.cc:288] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [CANDIDATE]: Term 1 election: Requested vote from peers 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987), d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403)
I20260926 12:38:28.147148 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8c7308d96047dba37e588a5bdbdc6d" candidate_uuid: "b1c17095522c4defb17da62f3338b0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55f9c919de44a9783165a90d2215c3f"
I20260926 12:38:28.147197 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8c7308d96047dba37e588a5bdbdc6d" candidate_uuid: "b1c17095522c4defb17da62f3338b0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1034bdda96f24a99a0a95649b7632c2f"
I20260926 12:38:28.147256 13169 raft_consensus.cc:3037] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.147353 13029 raft_consensus.cc:3037] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.148363 13169 raft_consensus.cc:2454] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1c17095522c4defb17da62f3338b0b8 in term 1.
I20260926 12:38:28.148507 13029 raft_consensus.cc:2454] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1c17095522c4defb17da62f3338b0b8 in term 1.
I20260926 12:38:28.148679 13241 leader_election.cc:302] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1c17095522c4defb17da62f3338b0b8, d55f9c919de44a9783165a90d2215c3f; no voters:
I20260926 12:38:28.148929 13386 raft_consensus.cc:2781] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:28.149108 13386 raft_consensus.cc:686] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 1 LEADER]: Becoming Leader. State: Replica: b1c17095522c4defb17da62f3338b0b8, State: Running, Role: LEADER
I20260926 12:38:28.149458 13386 consensus_queue.cc:237] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } }
I20260926 12:38:28.152937 12871 catalog_manager.cc:5495] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1c17095522c4defb17da62f3338b0b8 (127.10.224.131). New cstate: current_term: 1 leader_uuid: "b1c17095522c4defb17da62f3338b0b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } health_report { overall_health: UNKNOWN } } }
W20260926 12:38:28.193509 13216 tablet.cc:2369] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:38:28.196702 13076 tablet.cc:2369] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 12:38:28 dist-test-slave-m3pj krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:28.202979 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.193493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.10.224.190:42819)
0926 12:38:28.193647 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.193652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.193917 (+ 265us) server_negotiation.cc:407] Connection header received
0926 12:38:28.194046 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.194049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.194101 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.194189 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:28.194906 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.195456 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.196270 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.196600 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.196707 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.199414 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:28.199418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:28.199419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:28.199441 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:28.200816 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:28.201361 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:28.201365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:28.201367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:28.201462 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:28.201781 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:28.201784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:28.201785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:28.202052 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:28.202131 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.202572 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.202782 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":55}
I20260926 12:38:28.202991 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.193409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42819 (local address 127.0.0.1:47500)
0926 12:38:28.193812 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.193836 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.193944 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.194237 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.194239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.194675 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:28.194772 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.194778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.195612 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.195615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.196168 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.196175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.196716 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.196719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.196850 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.197626 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:28.197673 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:28.199253 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:28.200949 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:28.200958 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:28.200964 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:28.201194 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:28.201563 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:28.201566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:28.201570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:28.201678 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:28.202232 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:28.202239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:28.202371 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.202460 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.202700 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1,"thread_start_us":158,"threads_started":1}
I20260926 12:38:28.207438 12871 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:47500:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:38:28.208458 12871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 12:38:28.212558 13356 tablet.cc:2369] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:28.214820 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 926ccd69e014480b85dec4e099590f0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6149074fd1b42d79947caa1fce10e81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:38:28.214781 13009 tablet_service.cc:1467] Processing CreateTablet for tablet 926ccd69e014480b85dec4e099590f0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6149074fd1b42d79947caa1fce10e81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:38:28.215482 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926ccd69e014480b85dec4e099590f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.215493 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926ccd69e014480b85dec4e099590f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.218277 13379 tablet_bootstrap.cc:492] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: Bootstrap starting.
I20260926 12:38:28.218750 13378 tablet_bootstrap.cc:492] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: Bootstrap starting.
I20260926 12:38:28.218706 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 926ccd69e014480b85dec4e099590f0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6149074fd1b42d79947caa1fce10e81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:38:28.219318 13379 tablet_bootstrap.cc:654] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.219493 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926ccd69e014480b85dec4e099590f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.219761 13378 tablet_bootstrap.cc:654] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.220726 13379 tablet_bootstrap.cc:492] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: No bootstrap required, opened a new log
I20260926 12:38:28.220832 13379 ts_tablet_manager.cc:1397] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 12:38:28.221350 13379 raft_consensus.cc:348] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.221478 13379 raft_consensus.cc:374] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.221518 13379 raft_consensus.cc:729] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1034bdda96f24a99a0a95649b7632c2f, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.221686 13379 consensus_queue.cc:260] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.221987 13379 ts_tablet_manager.cc:1428] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260926 12:38:28.222009 13384 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.222167 13384 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:28.222550 13380 tablet_bootstrap.cc:492] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: Bootstrap starting.
I20260926 12:38:28.222929 13378 tablet_bootstrap.cc:492] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: No bootstrap required, opened a new log
I20260926 12:38:28.223047 13378 ts_tablet_manager.cc:1397] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 12:38:28.223598 13380 tablet_bootstrap.cc:654] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.223861 13378 raft_consensus.cc:348] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.223961 13378 raft_consensus.cc:374] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.223986 13378 raft_consensus.cc:729] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55f9c919de44a9783165a90d2215c3f, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.224151 13378 consensus_queue.cc:260] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.224560 13378 ts_tablet_manager.cc:1428] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:28.224495 13385 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.224710 13385 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:28.225270 13380 tablet_bootstrap.cc:492] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: No bootstrap required, opened a new log
I20260926 12:38:28.225380 13380 ts_tablet_manager.cc:1397] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:38:28.225860 13380 raft_consensus.cc:348] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.225973 13380 raft_consensus.cc:374] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.226013 13380 raft_consensus.cc:729] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c17095522c4defb17da62f3338b0b8, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.226159 13380 consensus_queue.cc:260] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.226444 13380 ts_tablet_manager.cc:1428] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:38:28.226477 13386 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.226622 13386 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:28.332437 13385 raft_consensus.cc:482] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:28.332599 13385 raft_consensus.cc:504] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.333811 13385 leader_election.cc:288] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987), b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963)
I20260926 12:38:28.338073 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.334341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:51913 (local address 127.10.224.129:40987)
0926 12:38:28.334589 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.334594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.334789 (+ 195us) server_negotiation.cc:407] Connection header received
0926 12:38:28.334949 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.334952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.335002 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.335094 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.335349 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.335911 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.337021 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.337435 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.337580 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.337635 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.337862 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.337936 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260926 12:38:28.338073 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.334420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:56583 (local address 127.10.224.131:39963)
0926 12:38:28.334565 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.334571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.334650 (+ 79us) server_negotiation.cc:407] Connection header received
0926 12:38:28.334839 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.334842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.334913 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.335003 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.335286 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.335819 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.337004 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.337405 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.337557 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.337618 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.337844 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.337928 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:38:28.338099 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.334201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:39963 (local address 127.10.224.130:56583)
0926 12:38:28.334564 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.334579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.334708 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.335017 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.335020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.335036 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.335173 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.335179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.335939 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.335945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.336877 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.336886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.337559 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.337562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.337694 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.337729 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.337820 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.337929 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":87,"thread_start_us":92,"threads_started":1}
I20260926 12:38:28.338104 13400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.334201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40987 (local address 127.10.224.130:51913)
0926 12:38:28.334690 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.334701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.334827 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.335104 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.335107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.335123 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.335237 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.335242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.336021 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.336025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.336892 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.336902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.337580 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.337584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.337708 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.337746 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.337836 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.337928 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":207,"thread_start_us":63,"threads_started":1}
I20260926 12:38:28.338649 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926ccd69e014480b85dec4e099590f0f" candidate_uuid: "d55f9c919de44a9783165a90d2215c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1034bdda96f24a99a0a95649b7632c2f" is_pre_election: true
I20260926 12:38:28.338816 13029 raft_consensus.cc:2454] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55f9c919de44a9783165a90d2215c3f in term 0.
I20260926 12:38:28.338832 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926ccd69e014480b85dec4e099590f0f" candidate_uuid: "d55f9c919de44a9783165a90d2215c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c17095522c4defb17da62f3338b0b8" is_pre_election: true
I20260926 12:38:28.339085 13309 raft_consensus.cc:2454] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55f9c919de44a9783165a90d2215c3f in term 0.
I20260926 12:38:28.339195 13103 leader_election.cc:302] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1034bdda96f24a99a0a95649b7632c2f, d55f9c919de44a9783165a90d2215c3f; no voters:
I20260926 12:38:28.339537 13385 raft_consensus.cc:2781] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:28.339604 13385 raft_consensus.cc:482] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:28.339653 13385 raft_consensus.cc:3037] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.340646 13385 raft_consensus.cc:504] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.341012 13385 leader_election.cc:288] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [CANDIDATE]: Term 1 election: Requested vote from peers 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987), b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963)
I20260926 12:38:28.341347 13029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926ccd69e014480b85dec4e099590f0f" candidate_uuid: "d55f9c919de44a9783165a90d2215c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1034bdda96f24a99a0a95649b7632c2f"
I20260926 12:38:28.341387 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926ccd69e014480b85dec4e099590f0f" candidate_uuid: "d55f9c919de44a9783165a90d2215c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c17095522c4defb17da62f3338b0b8"
I20260926 12:38:28.341490 13309 raft_consensus.cc:3037] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.341490 13029 raft_consensus.cc:3037] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.342722 13309 raft_consensus.cc:2454] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55f9c919de44a9783165a90d2215c3f in term 1.
I20260926 12:38:28.342804 13029 raft_consensus.cc:2454] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55f9c919de44a9783165a90d2215c3f in term 1.
I20260926 12:38:28.343189 13103 leader_election.cc:302] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1034bdda96f24a99a0a95649b7632c2f, d55f9c919de44a9783165a90d2215c3f; no voters:
I20260926 12:38:28.343400 13385 raft_consensus.cc:2781] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:28.343690 13385 raft_consensus.cc:686] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [term 1 LEADER]: Becoming Leader. State: Replica: d55f9c919de44a9783165a90d2215c3f, State: Running, Role: LEADER
I20260926 12:38:28.344072 13385 consensus_queue.cc:237] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.346462 13401 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: TxnStatusTablet state changed. Reason: New leader d55f9c919de44a9783165a90d2215c3f. Latest consensus state: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.346580 13401 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:38:28.346865 13403 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:38:28.347090 13403 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:38:28.347647 12870 catalog_manager.cc:5495] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d55f9c919de44a9783165a90d2215c3f (127.10.224.130). New cstate: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } health_report { overall_health: UNKNOWN } } }
I20260926 12:38:28.380503 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.375919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:38403 (local address 127.0.0.1:53724)
0926 12:38:28.376219 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.376235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.376359 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.376680 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.376683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.376691 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:28.376804 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.376818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.377644 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.377647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.378617 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.378627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.379069 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.379073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.379172 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.379228 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.380115 (+ 887us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.380141 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.380210 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.380336 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260926 12:38:28.381091 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.376040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.10.224.130:38403)
0926 12:38:28.376197 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.376201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.376355 (+ 154us) server_negotiation.cc:407] Connection header received
0926 12:38:28.376494 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.376496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.376554 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.376644 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:28.376947 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.377518 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.378743 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.378955 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.379037 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.379406 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.379998 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.380142 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.380928 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.380994 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 12:38:28.386307 13029 raft_consensus.cc:1261] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Refusing update from remote peer d55f9c919de44a9783165a90d2215c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:28.386305 13309 raft_consensus.cc:1261] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Refusing update from remote peer d55f9c919de44a9783165a90d2215c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:28.386997 13401 consensus_queue.cc:1035] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:28.387269 13385 consensus_queue.cc:1035] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:28.390908 13386 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: TxnStatusTablet state changed. Reason: New leader d55f9c919de44a9783165a90d2215c3f. Latest consensus state: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.391040 13386 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:28.392732 13384 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: TxnStatusTablet state changed. Reason: New leader d55f9c919de44a9783165a90d2215c3f. Latest consensus state: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.392863 13384 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:28.393658 13401 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.393778 13401 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:38:28.393977 13401 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.394064 13401 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P d55f9c919de44a9783165a90d2215c3f: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:38:28.394088 13386 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.394189 13386 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P b1c17095522c4defb17da62f3338b0b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:28.394742 13406 mvcc.cc:205] Tried to move back new op lower bound from 7333586159143526400 to 7333586158980300800. Current Snapshot: MvccSnapshot[applied={T|T < 7333586159143526400}]
I20260926 12:38:28.394920 13408 mvcc.cc:205] Tried to move back new op lower bound from 7333586159143526400 to 7333586158980300800. Current Snapshot: MvccSnapshot[applied={T|T < 7333586159143526400}]
I20260926 12:38:28.395028 13384 tablet_replica.cc:439] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } } }
I20260926 12:38:28.395145 13384 tablet_replica.cc:442] T 926ccd69e014480b85dec4e099590f0f P 1034bdda96f24a99a0a95649b7632c2f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:38:28.397008 13407 mvcc.cc:205] Tried to move back new op lower bound from 7333586159143526400 to 7333586158980300800. Current Snapshot: MvccSnapshot[applied={T|T < 7333586159143526400}]
I20260926 12:38:28.405545 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.401177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.10.224.131:39963)
0926 12:38:28.401307 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.401312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.401349 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:38:28.401530 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.401533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.401594 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.401702 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:28.401941 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.402720 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.403860 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.404213 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.404311 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.404564 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.405096 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.405208 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.405348 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.405404 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:38:28.417394 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.413953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:36123 (local address 127.10.224.130:38403)
0926 12:38:28.414088 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.414091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.414107 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:38:28.414181 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.414183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.414237 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.414304 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:28.414568 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.415062 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.416254 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.416530 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.416615 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.416957 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.417023 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.417077 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.417193 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.417237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:38:28.417392 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.413855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:38403 (local address 127.10.224.131:36123)
0926 12:38:28.413998 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.414015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.414105 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.414338 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.414341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.414349 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:28.414465 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.414471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.415161 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.415164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.416147 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.416158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.416642 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.416645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.416742 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.416794 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.417103 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.417109 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.417174 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.417254 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 12:38:28.424610 13029 raft_consensus.cc:1261] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Refusing update from remote peer b1c17095522c4defb17da62f3338b0b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:28.424748 13169 raft_consensus.cc:1261] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Refusing update from remote peer b1c17095522c4defb17da62f3338b0b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:28.425175 13388 consensus_queue.cc:1035] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:28.425688 13388 consensus_queue.cc:1035] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:28.461354 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.456610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:41511 (local address 127.10.224.131:39963)
0926 12:38:28.456863 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.456868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.457126 (+ 258us) server_negotiation.cc:407] Connection header received
0926 12:38:28.457328 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.457331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.457384 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.457431 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:28.458035 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.458745 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.459980 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.460306 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.460400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.460859 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.460948 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.460996 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.461173 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.461230 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260926 12:38:28.461390 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.456508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:39963 (local address 127.10.224.130:41511)
0926 12:38:28.457050 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.457064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.457207 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.457569 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.457572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.457580 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:28.457676 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.457681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.458901 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.458904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.459842 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.459855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.460524 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.460527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.460610 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.460658 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.461057 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.461063 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.461129 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.461213 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":125,"threads_started":1}
I20260926 12:38:28.495434 12870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:28.495690 12870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:28.498768 12870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: Sending DeleteTablet for 3 replicas of tablet de8c7308d96047dba37e588a5bdbdc6d
I20260926 12:38:28.499653 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet de8c7308d96047dba37e588a5bdbdc6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:28 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:35254
I20260926 12:38:28.499776 13009 tablet_service.cc:1514] Processing DeleteTablet for tablet de8c7308d96047dba37e588a5bdbdc6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:28 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:60346
I20260926 12:38:28.500492 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet de8c7308d96047dba37e588a5bdbdc6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:28 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:53720
I20260926 12:38:28.501121 13425 tablet_replica.cc:330] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: stopping tablet replica
I20260926 12:38:28.501117 12870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:38:28.501591 13427 tablet_replica.cc:330] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: stopping tablet replica
W20260926 12:38:28.501592 12870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:28.502010 13427 raft_consensus.cc:2229] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:28.502391 13425 raft_consensus.cc:2229] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:28.502672 13427 raft_consensus.cc:2258] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:28.503248 13426 tablet_replica.cc:330] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: stopping tablet replica
I20260926 12:38:28.503575 13426 raft_consensus.cc:2229] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:28.503782 13426 raft_consensus.cc:2258] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:28.507457 13009 tablet_service.cc:1467] Processing CreateTablet for tablet 756a0c30826c4d03a857874880d9b722 (DEFAULT_TABLE table=test-table [id=3477d2dbe7744e08b36964b7634b5899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:28.507813 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756a0c30826c4d03a857874880d9b722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.508190 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 756a0c30826c4d03a857874880d9b722 (DEFAULT_TABLE table=test-table [id=3477d2dbe7744e08b36964b7634b5899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:28.508495 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756a0c30826c4d03a857874880d9b722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.511332 13379 tablet_bootstrap.cc:492] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: Bootstrap starting.
I20260926 12:38:28.511543 13380 tablet_bootstrap.cc:492] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: Bootstrap starting.
I20260926 12:38:28.512316 13379 tablet_bootstrap.cc:654] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.512439 13380 tablet_bootstrap.cc:654] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.507843 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 756a0c30826c4d03a857874880d9b722 (DEFAULT_TABLE table=test-table [id=3477d2dbe7744e08b36964b7634b5899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:28.513581 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 756a0c30826c4d03a857874880d9b722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:28.514168 13426 ts_tablet_manager.cc:1905] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:28.516360 13378 tablet_bootstrap.cc:492] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: Bootstrap starting.
I20260926 12:38:28.517405 13378 tablet_bootstrap.cc:654] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:28.518374 13426 ts_tablet_manager.cc:1918] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:28.518491 13426 log.cc:1196] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/de8c7308d96047dba37e588a5bdbdc6d
I20260926 12:38:28.518707 13425 raft_consensus.cc:2258] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:28.518822 13426 ts_tablet_manager.cc:1939] T de8c7308d96047dba37e588a5bdbdc6d P 1034bdda96f24a99a0a95649b7632c2f: Deleting consensus metadata
I20260926 12:38:28.520371 13427 ts_tablet_manager.cc:1905] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:28.520401 12859 catalog_manager.cc:4841] TS 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987): tablet de8c7308d96047dba37e588a5bdbdc6d (table test-table [id=ec427583c8754b0e8ca3af95155bcf91]) successfully deleted
I20260926 12:38:28.521924 13425 ts_tablet_manager.cc:1905] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:28.522207 13379 tablet_bootstrap.cc:492] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: No bootstrap required, opened a new log
I20260926 12:38:28.522308 13379 ts_tablet_manager.cc:1397] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260926 12:38:28.522786 13379 raft_consensus.cc:348] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.522912 13379 raft_consensus.cc:374] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.522950 13379 raft_consensus.cc:729] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1034bdda96f24a99a0a95649b7632c2f, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.523070 13379 consensus_queue.cc:260] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.523327 13379 ts_tablet_manager.cc:1428] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:28.528275 13425 ts_tablet_manager.cc:1918] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:28.528482 13425 log.cc:1196] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/de8c7308d96047dba37e588a5bdbdc6d
I20260926 12:38:28.529013 13425 ts_tablet_manager.cc:1939] T de8c7308d96047dba37e588a5bdbdc6d P b1c17095522c4defb17da62f3338b0b8: Deleting consensus metadata
I20260926 12:38:28.529083 13427 ts_tablet_manager.cc:1918] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:28.529196 13427 log.cc:1196] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/de8c7308d96047dba37e588a5bdbdc6d
I20260926 12:38:28.529511 13427 ts_tablet_manager.cc:1939] T de8c7308d96047dba37e588a5bdbdc6d P d55f9c919de44a9783165a90d2215c3f: Deleting consensus metadata
I20260926 12:38:28.530534 12857 catalog_manager.cc:4841] TS b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963): tablet de8c7308d96047dba37e588a5bdbdc6d (table test-table [id=ec427583c8754b0e8ca3af95155bcf91]) successfully deleted
I20260926 12:38:28.530866 12857 catalog_manager.cc:4841] TS d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403): tablet de8c7308d96047dba37e588a5bdbdc6d (table test-table [id=ec427583c8754b0e8ca3af95155bcf91]) successfully deleted
I20260926 12:38:28.531126 13378 tablet_bootstrap.cc:492] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: No bootstrap required, opened a new log
I20260926 12:38:28.531215 13378 ts_tablet_manager.cc:1397] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260926 12:38:28.531674 13378 raft_consensus.cc:348] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.531792 13378 raft_consensus.cc:374] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.531823 13378 raft_consensus.cc:729] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55f9c919de44a9783165a90d2215c3f, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.531942 13378 consensus_queue.cc:260] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.532203 13378 ts_tablet_manager.cc:1428] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:28.533160 13380 tablet_bootstrap.cc:492] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: No bootstrap required, opened a new log
I20260926 12:38:28.533252 13380 ts_tablet_manager.cc:1397] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20260926 12:38:28.533720 13380 raft_consensus.cc:348] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.533833 13380 raft_consensus.cc:374] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:28.533864 13380 raft_consensus.cc:729] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c17095522c4defb17da62f3338b0b8, State: Initialized, Role: FOLLOWER
I20260926 12:38:28.533975 13380 consensus_queue.cc:260] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.534209 13380 ts_tablet_manager.cc:1428] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:28.771310 13384 raft_consensus.cc:482] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:28.771492 13384 raft_consensus.cc:504] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.772682 13384 leader_election.cc:288] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403), b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963)
I20260926 12:38:28.776986 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.773587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:38279 (local address 127.10.224.131:39963)
0926 12:38:28.773756 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.773761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.773776 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:28.773918 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.773921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.773968 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.774063 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.774341 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.774842 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.775943 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.776322 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.776486 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.776541 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.776758 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.776836 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:38:28.777000 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.773117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:53577 (local address 127.10.224.130:38403)
0926 12:38:28.773249 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.773253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.773451 (+ 198us) server_negotiation.cc:407] Connection header received
0926 12:38:28.773706 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.773710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.773773 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.773854 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:28.774170 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.774745 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.775886 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.776302 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.776487 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.776543 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.776810 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.776886 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:38:28.777021 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.773454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:39963 (local address 127.10.224.129:38279)
0926 12:38:28.773693 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.773707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.773805 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.774081 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.774083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.774097 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.774237 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.774243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.774968 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.774971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.775792 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.775801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.776478 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.776481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.776629 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.776652 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.776712 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.776804 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":26,"thread_start_us":59,"threads_started":1}
I20260926 12:38:28.777027 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.773010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:38403 (local address 127.10.224.129:53577)
0926 12:38:28.773323 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.773341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.773540 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.773886 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.773892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.773912 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:28.774056 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.774064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.774863 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.774869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.775762 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.775771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.776499 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.776503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.776656 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.776674 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.776769 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.776836 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":40,"thread_start_us":112,"threads_started":1}
I20260926 12:38:28.777661 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756a0c30826c4d03a857874880d9b722" candidate_uuid: "1034bdda96f24a99a0a95649b7632c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55f9c919de44a9783165a90d2215c3f" is_pre_election: true
I20260926 12:38:28.777660 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756a0c30826c4d03a857874880d9b722" candidate_uuid: "1034bdda96f24a99a0a95649b7632c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c17095522c4defb17da62f3338b0b8" is_pre_election: true
I20260926 12:38:28.777822 13169 raft_consensus.cc:2454] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1034bdda96f24a99a0a95649b7632c2f in term 0.
I20260926 12:38:28.777825 13309 raft_consensus.cc:2454] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1034bdda96f24a99a0a95649b7632c2f in term 0.
I20260926 12:38:28.778182 12961 leader_election.cc:302] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1034bdda96f24a99a0a95649b7632c2f, b1c17095522c4defb17da62f3338b0b8; no voters:
I20260926 12:38:28.778466 13384 raft_consensus.cc:2781] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:28.778534 13384 raft_consensus.cc:482] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:28.778573 13384 raft_consensus.cc:3037] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.780049 13384 raft_consensus.cc:504] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.780408 13384 leader_election.cc:288] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [CANDIDATE]: Term 1 election: Requested vote from peers d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403), b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963)
I20260926 12:38:28.780778 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756a0c30826c4d03a857874880d9b722" candidate_uuid: "1034bdda96f24a99a0a95649b7632c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55f9c919de44a9783165a90d2215c3f"
I20260926 12:38:28.780778 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "756a0c30826c4d03a857874880d9b722" candidate_uuid: "1034bdda96f24a99a0a95649b7632c2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c17095522c4defb17da62f3338b0b8"
I20260926 12:38:28.780908 13309 raft_consensus.cc:3037] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.780908 13169 raft_consensus.cc:3037] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:28.782035 13169 raft_consensus.cc:2454] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1034bdda96f24a99a0a95649b7632c2f in term 1.
I20260926 12:38:28.782035 13309 raft_consensus.cc:2454] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1034bdda96f24a99a0a95649b7632c2f in term 1.
I20260926 12:38:28.782303 12961 leader_election.cc:302] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1034bdda96f24a99a0a95649b7632c2f, d55f9c919de44a9783165a90d2215c3f; no voters:
I20260926 12:38:28.782476 13384 raft_consensus.cc:2781] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:28.782676 13384 raft_consensus.cc:686] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 1 LEADER]: Becoming Leader. State: Replica: 1034bdda96f24a99a0a95649b7632c2f, State: Running, Role: LEADER
I20260926 12:38:28.782958 13384 consensus_queue.cc:237] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:28.785351 12870 catalog_manager.cc:5495] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129). New cstate: current_term: 1 leader_uuid: "1034bdda96f24a99a0a95649b7632c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } health_report { overall_health: UNKNOWN } } }
I20260926 12:38:28.814416 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.810816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.10.224.129:40987)
0926 12:38:28.810966 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.810969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.810983 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:38:28.811091 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.811093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.811146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.811236 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:28.811543 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.812056 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.812906 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.813172 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.813282 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.813547 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.814017 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.814148 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.814193 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.814246 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:38:28.825253 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.821797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:49623 (local address 127.10.224.130:38403)
0926 12:38:28.821968 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.821973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.822146 (+ 173us) server_negotiation.cc:407] Connection header received
0926 12:38:28.822316 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.822319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.822375 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.822499 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:28.822765 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.823275 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.824175 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.824406 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.824518 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.824808 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.824878 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.824953 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.825046 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.825104 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 12:38:28.825265 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.821623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:38403 (local address 127.10.224.129:49623)
0926 12:38:28.822055 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.822068 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.822203 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.822511 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.822514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.822522 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:28.822656 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.822661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.823390 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.823394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.824048 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.824057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.824518 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.824521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.824624 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.824667 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.824968 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.824977 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.825037 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.825119 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
I20260926 12:38:28.831588 13309 raft_consensus.cc:1261] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Refusing update from remote peer 1034bdda96f24a99a0a95649b7632c2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:28.831635 13169 raft_consensus.cc:1261] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Refusing update from remote peer 1034bdda96f24a99a0a95649b7632c2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:28.832101 13384 consensus_queue.cc:1035] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:28.832322 13433 consensus_queue.cc:1035] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:28.851835 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.848228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:46495 (local address 127.10.224.129:40987)
0926 12:38:28.848363 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:28.848367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:28.848378 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:38:28.848416 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:28.848419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:28.848458 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:28.848522 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:28.848810 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.849525 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.850499 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.850837 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.850979 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.851216 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.851327 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.851408 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:28.851638 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 12:38:28.851695 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:38:28.851835 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:28.848040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40987 (local address 127.10.224.130:46495)
0926 12:38:28.848158 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:28.848171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:28.848241 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:28.848558 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:28.848562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:28.848571 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:28.848673 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.848678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.849637 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.849643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.850368 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:28.850378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:28.850956 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:28.850960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:28.851053 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:28.851091 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:28.851426 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:28.851452 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:38:28.851598 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 12:38:28.851706 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 12:38:28.852767 13407 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 12:38:30.412935 12870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:30.413152 12870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:30.415869 12870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: Sending DeleteTablet for 3 replicas of tablet 756a0c30826c4d03a857874880d9b722
I20260926 12:38:30.416484 13009 tablet_service.cc:1514] Processing DeleteTablet for tablet 756a0c30826c4d03a857874880d9b722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:30 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:60346
I20260926 12:38:30.416882 13465 tablet_replica.cc:330] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: stopping tablet replica
I20260926 12:38:30.416867 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 756a0c30826c4d03a857874880d9b722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:30 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:53720
I20260926 12:38:30.417013 13465 raft_consensus.cc:2229] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:30.417213 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 756a0c30826c4d03a857874880d9b722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:30 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:35254
I20260926 12:38:30.417382 13465 raft_consensus.cc:2258] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:30.417776 13467 tablet_replica.cc:330] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: stopping tablet replica
I20260926 12:38:30.417892 13467 raft_consensus.cc:2229] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:30.418013 13467 raft_consensus.cc:2258] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:30.418025 13465 ts_tablet_manager.cc:1905] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:30.418340 13466 tablet_replica.cc:330] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: stopping tablet replica
I20260926 12:38:30.418475 13466 raft_consensus.cc:2229] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:30.418591 13466 raft_consensus.cc:2258] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:30.419282 13467 ts_tablet_manager.cc:1905] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:30.420116 13466 ts_tablet_manager.cc:1905] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:30.421201 13465 ts_tablet_manager.cc:1918] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:38:30.421305 13465 log.cc:1196] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/756a0c30826c4d03a857874880d9b722
I20260926 12:38:30.421711 13465 ts_tablet_manager.cc:1939] T 756a0c30826c4d03a857874880d9b722 P 1034bdda96f24a99a0a95649b7632c2f: Deleting consensus metadata
I20260926 12:38:30.422191 13467 ts_tablet_manager.cc:1918] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:38:30.422276 13467 log.cc:1196] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/756a0c30826c4d03a857874880d9b722
I20260926 12:38:30.422580 12859 catalog_manager.cc:4841] TS 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987): tablet 756a0c30826c4d03a857874880d9b722 (table test-table [id=3477d2dbe7744e08b36964b7634b5899]) successfully deleted
I20260926 12:38:30.422696 13467 ts_tablet_manager.cc:1939] T 756a0c30826c4d03a857874880d9b722 P b1c17095522c4defb17da62f3338b0b8: Deleting consensus metadata
I20260926 12:38:30.423338 12857 catalog_manager.cc:4841] TS b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963): tablet 756a0c30826c4d03a857874880d9b722 (table test-table [id=3477d2dbe7744e08b36964b7634b5899]) successfully deleted
I20260926 12:38:30.423456 13466 ts_tablet_manager.cc:1918] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:38:30.423579 13466 log.cc:1196] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/756a0c30826c4d03a857874880d9b722
I20260926 12:38:30.423882 13466 ts_tablet_manager.cc:1939] T 756a0c30826c4d03a857874880d9b722 P d55f9c919de44a9783165a90d2215c3f: Deleting consensus metadata
I20260926 12:38:30.424629 12857 catalog_manager.cc:4841] TS d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403): tablet 756a0c30826c4d03a857874880d9b722 (table test-table [id=3477d2dbe7744e08b36964b7634b5899]) successfully deleted
Sep 26 12:38:30 dist-test-slave-m3pj krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:38:30 dist-test-slave-m3pj krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:30.448172 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:30.438354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.10.224.190:42819)
0926 12:38:30.438768 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:30.438774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:30.438789 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:30.438834 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:30.438837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:30.438879 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:30.438970 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:30.439737 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.440301 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.441262 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.441504 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.441593 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:30.444404 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:30.444408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:30.444410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:30.444438 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:30.446122 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:30.446690 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:30.446697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:30.446699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:30.446795 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:30.447134 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:30.447139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:30.447141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:30.447355 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:30.447511 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:30.447831 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 12:38:30.447980 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":250,"threads_started":1}
I20260926 12:38:30.451413 12870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:30.451974 12870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:30.457146 13009 tablet_service.cc:1467] Processing CreateTablet for tablet 829458da47aa4583aea54518e4fc767f (DEFAULT_TABLE table=test-table [id=f838fee2a72d4fd1bdfb62cbe9b78bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:30.457170 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 829458da47aa4583aea54518e4fc767f (DEFAULT_TABLE table=test-table [id=f838fee2a72d4fd1bdfb62cbe9b78bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:30.457244 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 829458da47aa4583aea54518e4fc767f (DEFAULT_TABLE table=test-table [id=f838fee2a72d4fd1bdfb62cbe9b78bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:30.457475 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829458da47aa4583aea54518e4fc767f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:30.457475 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829458da47aa4583aea54518e4fc767f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:30.457569 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829458da47aa4583aea54518e4fc767f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:30.460467 13477 tablet_bootstrap.cc:492] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: Bootstrap starting.
I20260926 12:38:30.460467 13476 tablet_bootstrap.cc:492] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: Bootstrap starting.
I20260926 12:38:30.460651 13478 tablet_bootstrap.cc:492] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: Bootstrap starting.
I20260926 12:38:30.461481 13477 tablet_bootstrap.cc:654] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:30.461481 13476 tablet_bootstrap.cc:654] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:30.461493 13478 tablet_bootstrap.cc:654] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:30.462709 13477 tablet_bootstrap.cc:492] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: No bootstrap required, opened a new log
I20260926 12:38:30.462818 13477 ts_tablet_manager.cc:1397] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:38:30.462913 13478 tablet_bootstrap.cc:492] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: No bootstrap required, opened a new log
I20260926 12:38:30.462999 13478 ts_tablet_manager.cc:1397] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:38:30.463277 13477 raft_consensus.cc:348] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.463375 13477 raft_consensus.cc:374] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:30.463398 13477 raft_consensus.cc:729] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1034bdda96f24a99a0a95649b7632c2f, State: Initialized, Role: FOLLOWER
I20260926 12:38:30.463354 13478 raft_consensus.cc:348] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.463498 13478 raft_consensus.cc:374] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:30.463572 13478 raft_consensus.cc:729] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c17095522c4defb17da62f3338b0b8, State: Initialized, Role: FOLLOWER
I20260926 12:38:30.463568 13477 consensus_queue.cc:260] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.463714 13478 consensus_queue.cc:260] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.463873 13477 ts_tablet_manager.cc:1428] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:30.464063 13476 tablet_bootstrap.cc:492] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: No bootstrap required, opened a new log
I20260926 12:38:30.464098 13478 ts_tablet_manager.cc:1428] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:30.464159 13476 ts_tablet_manager.cc:1397] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 12:38:30.464602 13476 raft_consensus.cc:348] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.464783 13476 raft_consensus.cc:374] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:30.464828 13476 raft_consensus.cc:729] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55f9c919de44a9783165a90d2215c3f, State: Initialized, Role: FOLLOWER
I20260926 12:38:30.464982 13476 consensus_queue.cc:260] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.465263 13476 ts_tablet_manager.cc:1428] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:30.594666 13446 raft_consensus.cc:482] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:30.594848 13446 raft_consensus.cc:504] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.595319 13446 leader_election.cc:288] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987), b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963)
I20260926 12:38:30.595866 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829458da47aa4583aea54518e4fc767f" candidate_uuid: "d55f9c919de44a9783165a90d2215c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c17095522c4defb17da62f3338b0b8" is_pre_election: true
I20260926 12:38:30.595907 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829458da47aa4583aea54518e4fc767f" candidate_uuid: "d55f9c919de44a9783165a90d2215c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1034bdda96f24a99a0a95649b7632c2f" is_pre_election: true
I20260926 12:38:30.596035 13308 raft_consensus.cc:2454] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55f9c919de44a9783165a90d2215c3f in term 0.
I20260926 12:38:30.596091 13028 raft_consensus.cc:2454] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55f9c919de44a9783165a90d2215c3f in term 0.
I20260926 12:38:30.596477 13101 leader_election.cc:302] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1c17095522c4defb17da62f3338b0b8, d55f9c919de44a9783165a90d2215c3f; no voters:
I20260926 12:38:30.596688 13446 raft_consensus.cc:2781] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:30.596769 13446 raft_consensus.cc:482] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:30.596848 13446 raft_consensus.cc:3037] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:30.598206 13446 raft_consensus.cc:504] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.598621 13446 leader_election.cc:288] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [CANDIDATE]: Term 1 election: Requested vote from peers 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987), b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963)
I20260926 12:38:30.599030 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829458da47aa4583aea54518e4fc767f" candidate_uuid: "d55f9c919de44a9783165a90d2215c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1034bdda96f24a99a0a95649b7632c2f"
I20260926 12:38:30.599030 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829458da47aa4583aea54518e4fc767f" candidate_uuid: "d55f9c919de44a9783165a90d2215c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1c17095522c4defb17da62f3338b0b8"
I20260926 12:38:30.599179 13308 raft_consensus.cc:3037] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:30.599154 13028 raft_consensus.cc:3037] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:30.600241 13308 raft_consensus.cc:2454] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55f9c919de44a9783165a90d2215c3f in term 1.
I20260926 12:38:30.600270 13028 raft_consensus.cc:2454] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55f9c919de44a9783165a90d2215c3f in term 1.
I20260926 12:38:30.600584 13101 leader_election.cc:302] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1c17095522c4defb17da62f3338b0b8, d55f9c919de44a9783165a90d2215c3f; no voters:
I20260926 12:38:30.600800 13446 raft_consensus.cc:2781] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:30.600875 13446 raft_consensus.cc:686] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 1 LEADER]: Becoming Leader. State: Replica: d55f9c919de44a9783165a90d2215c3f, State: Running, Role: LEADER
I20260926 12:38:30.601030 13446 consensus_queue.cc:237] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.602763 12872 catalog_manager.cc:5495] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d55f9c919de44a9783165a90d2215c3f (127.10.224.130). New cstate: current_term: 1 leader_uuid: "d55f9c919de44a9783165a90d2215c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } health_report { overall_health: UNKNOWN } } }
I20260926 12:38:30.626891 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:30.623455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.10.224.130:38403)
0926 12:38:30.623808 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:30.623818 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:30.623830 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:38:30.623879 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:30.623881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:30.623922 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:30.624011 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:30.624308 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.624849 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.625758 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.625998 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.626113 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:30.626386 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:30.626473 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:30.626565 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:30.626661 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 12:38:30.626718 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":166,"threads_started":1}
I20260926 12:38:30.637198 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:30.633626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:38403 (local address 127.10.224.130:51997)
0926 12:38:30.633947 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:30.633963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:30.634106 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:30.634347 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:30.634350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:30.634357 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:38:30.634504 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:30.634509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.635256 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.635260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:30.636016 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:30.636025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.636462 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.636465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:30.636548 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:30.636578 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:38:30.636868 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:38:30.636875 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:38:30.636938 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 12:38:30.637018 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":22,"thread_start_us":127,"threads_started":1}
I20260926 12:38:30.637198 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:30.633809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:51997 (local address 127.10.224.130:38403)
0926 12:38:30.633946 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:30.633950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:30.634048 (+ 98us) server_negotiation.cc:407] Connection header received
0926 12:38:30.634202 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:30.634206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:30.634257 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:30.634339 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:30.634612 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.635117 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.636138 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.636349 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.636460 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:30.636692 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:30.636768 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:30.636862 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:30.636963 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 12:38:30.637017 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":56}
I20260926 12:38:30.643055 13308 raft_consensus.cc:1261] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Refusing update from remote peer d55f9c919de44a9783165a90d2215c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:30.643198 13028 raft_consensus.cc:1261] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Refusing update from remote peer d55f9c919de44a9783165a90d2215c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:30.643603 13484 consensus_queue.cc:1035] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:30.643779 13446 consensus_queue.cc:1035] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:30.679790 12871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:30.679966 12871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:30.682262 12871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: Sending DeleteTablet for 3 replicas of tablet 829458da47aa4583aea54518e4fc767f
I20260926 12:38:30.682899 13009 tablet_service.cc:1514] Processing DeleteTablet for tablet 829458da47aa4583aea54518e4fc767f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:30 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:60346
I20260926 12:38:30.683014 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 829458da47aa4583aea54518e4fc767f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:30 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:35254
I20260926 12:38:30.683121 13465 tablet_replica.cc:330] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: stopping tablet replica
I20260926 12:38:30.683239 13465 raft_consensus.cc:2229] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:30.683223 13467 tablet_replica.cc:330] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: stopping tablet replica
I20260926 12:38:30.683346 13465 raft_consensus.cc:2258] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:30.683367 13467 raft_consensus.cc:2229] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:30.683485 13467 raft_consensus.cc:2258] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:30.683730 13148 tablet_service.cc:1514] Processing DeleteTablet for tablet 829458da47aa4583aea54518e4fc767f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:30 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:53720
I20260926 12:38:30.683913 13466 tablet_replica.cc:330] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: stopping tablet replica
I20260926 12:38:30.684033 13466 raft_consensus.cc:2229] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:30.684338 13465 ts_tablet_manager.cc:1905] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:30.684374 13466 raft_consensus.cc:2258] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:30.685088 13466 ts_tablet_manager.cc:1905] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:30.685530 12871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:30.685974 12871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:30.687191 13465 ts_tablet_manager.cc:1918] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:30.687260 13465 log.cc:1196] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/829458da47aa4583aea54518e4fc767f
I20260926 12:38:30.687244 13467 ts_tablet_manager.cc:1905] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:30.687589 13465 ts_tablet_manager.cc:1939] T 829458da47aa4583aea54518e4fc767f P 1034bdda96f24a99a0a95649b7632c2f: Deleting consensus metadata
I20260926 12:38:30.688671 12859 catalog_manager.cc:4841] TS 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987): tablet 829458da47aa4583aea54518e4fc767f (table test-table [id=f838fee2a72d4fd1bdfb62cbe9b78bd1]) successfully deleted
I20260926 12:38:30.689616 13466 ts_tablet_manager.cc:1918] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:30.689718 13466 log.cc:1196] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/829458da47aa4583aea54518e4fc767f
I20260926 12:38:30.689754 13467 ts_tablet_manager.cc:1918] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:38:30.689836 13467 log.cc:1196] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/829458da47aa4583aea54518e4fc767f
I20260926 12:38:30.690022 13466 ts_tablet_manager.cc:1939] T 829458da47aa4583aea54518e4fc767f P d55f9c919de44a9783165a90d2215c3f: Deleting consensus metadata
I20260926 12:38:30.690115 13467 ts_tablet_manager.cc:1939] T 829458da47aa4583aea54518e4fc767f P b1c17095522c4defb17da62f3338b0b8: Deleting consensus metadata
I20260926 12:38:30.690810 12857 catalog_manager.cc:4841] TS b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963): tablet 829458da47aa4583aea54518e4fc767f (table test-table [id=f838fee2a72d4fd1bdfb62cbe9b78bd1]) successfully deleted
I20260926 12:38:30.691107 12857 catalog_manager.cc:4841] TS d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403): tablet 829458da47aa4583aea54518e4fc767f (table test-table [id=f838fee2a72d4fd1bdfb62cbe9b78bd1]) successfully deleted
I20260926 12:38:30.694111 13009 tablet_service.cc:1467] Processing CreateTablet for tablet d0abe0f72b9a4c29b183145aadbda1ee (DEFAULT_TABLE table=test-table [id=a2c104b5e0b34c2e95a106fcd60c57b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:30.694375 13009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0abe0f72b9a4c29b183145aadbda1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:30.696460 13477 tablet_bootstrap.cc:492] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f: Bootstrap starting.
I20260926 12:38:30.696967 13148 tablet_service.cc:1467] Processing CreateTablet for tablet d0abe0f72b9a4c29b183145aadbda1ee (DEFAULT_TABLE table=test-table [id=a2c104b5e0b34c2e95a106fcd60c57b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:30.697186 13289 tablet_service.cc:1467] Processing CreateTablet for tablet d0abe0f72b9a4c29b183145aadbda1ee (DEFAULT_TABLE table=test-table [id=a2c104b5e0b34c2e95a106fcd60c57b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:30.697268 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0abe0f72b9a4c29b183145aadbda1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:30.697441 13477 tablet_bootstrap.cc:654] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:30.697470 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0abe0f72b9a4c29b183145aadbda1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:30.698904 13477 tablet_bootstrap.cc:492] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f: No bootstrap required, opened a new log
I20260926 12:38:30.699013 13477 ts_tablet_manager.cc:1397] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:38:30.699419 13477 raft_consensus.cc:348] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.699589 13477 raft_consensus.cc:374] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:30.699630 13477 raft_consensus.cc:729] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1034bdda96f24a99a0a95649b7632c2f, State: Initialized, Role: FOLLOWER
I20260926 12:38:30.699774 13477 consensus_queue.cc:260] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.699928 13478 tablet_bootstrap.cc:492] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: Bootstrap starting.
I20260926 12:38:30.699988 13476 tablet_bootstrap.cc:492] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: Bootstrap starting.
I20260926 12:38:30.700066 13477 ts_tablet_manager.cc:1428] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:30.700693 13478 tablet_bootstrap.cc:654] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:30.700894 13476 tablet_bootstrap.cc:654] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:30.702262 13478 tablet_bootstrap.cc:492] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: No bootstrap required, opened a new log
I20260926 12:38:30.702375 13478 ts_tablet_manager.cc:1397] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:38:30.702770 13478 raft_consensus.cc:348] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.702898 13478 raft_consensus.cc:374] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:30.702934 13478 raft_consensus.cc:729] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c17095522c4defb17da62f3338b0b8, State: Initialized, Role: FOLLOWER
I20260926 12:38:30.703063 13478 consensus_queue.cc:260] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.703318 13478 ts_tablet_manager.cc:1428] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:30.708539 13476 tablet_bootstrap.cc:492] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: No bootstrap required, opened a new log
I20260926 12:38:30.708711 13476 ts_tablet_manager.cc:1397] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260926 12:38:30.709172 13476 raft_consensus.cc:348] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.709303 13476 raft_consensus.cc:374] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:30.709343 13476 raft_consensus.cc:729] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55f9c919de44a9783165a90d2215c3f, State: Initialized, Role: FOLLOWER
I20260926 12:38:30.709482 13476 consensus_queue.cc:260] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.709769 13476 ts_tablet_manager.cc:1428] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:30.924948 13482 raft_consensus.cc:482] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:30.925177 13482 raft_consensus.cc:504] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.925719 13482 leader_election.cc:288] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987), d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403)
I20260926 12:38:30.926208 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0abe0f72b9a4c29b183145aadbda1ee" candidate_uuid: "b1c17095522c4defb17da62f3338b0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1034bdda96f24a99a0a95649b7632c2f" is_pre_election: true
I20260926 12:38:30.926245 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0abe0f72b9a4c29b183145aadbda1ee" candidate_uuid: "b1c17095522c4defb17da62f3338b0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55f9c919de44a9783165a90d2215c3f" is_pre_election: true
I20260926 12:38:30.926342 13028 raft_consensus.cc:2454] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1c17095522c4defb17da62f3338b0b8 in term 0.
I20260926 12:38:30.926452 13168 raft_consensus.cc:2454] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1c17095522c4defb17da62f3338b0b8 in term 0.
I20260926 12:38:30.926700 13243 leader_election.cc:302] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1034bdda96f24a99a0a95649b7632c2f, b1c17095522c4defb17da62f3338b0b8; no voters:
I20260926 12:38:30.926944 13482 raft_consensus.cc:2781] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:30.927021 13482 raft_consensus.cc:482] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:30.927075 13482 raft_consensus.cc:3037] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:30.928170 13482 raft_consensus.cc:504] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.928565 13482 leader_election.cc:288] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [CANDIDATE]: Term 1 election: Requested vote from peers 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987), d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403)
I20260926 12:38:30.929000 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0abe0f72b9a4c29b183145aadbda1ee" candidate_uuid: "b1c17095522c4defb17da62f3338b0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55f9c919de44a9783165a90d2215c3f"
I20260926 12:38:30.929075 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0abe0f72b9a4c29b183145aadbda1ee" candidate_uuid: "b1c17095522c4defb17da62f3338b0b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1034bdda96f24a99a0a95649b7632c2f"
I20260926 12:38:30.929149 13168 raft_consensus.cc:3037] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:30.929268 13028 raft_consensus.cc:3037] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:30.930240 13168 raft_consensus.cc:2454] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1c17095522c4defb17da62f3338b0b8 in term 1.
I20260926 12:38:30.930281 13028 raft_consensus.cc:2454] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1c17095522c4defb17da62f3338b0b8 in term 1.
I20260926 12:38:30.930619 13241 leader_election.cc:302] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1c17095522c4defb17da62f3338b0b8, d55f9c919de44a9783165a90d2215c3f; no voters:
I20260926 12:38:30.930810 13482 raft_consensus.cc:2781] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:30.931015 13482 raft_consensus.cc:686] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 1 LEADER]: Becoming Leader. State: Replica: b1c17095522c4defb17da62f3338b0b8, State: Running, Role: LEADER
I20260926 12:38:30.931175 13482 consensus_queue.cc:237] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } }
I20260926 12:38:30.932906 12871 catalog_manager.cc:5495] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1c17095522c4defb17da62f3338b0b8 (127.10.224.131). New cstate: current_term: 1 leader_uuid: "b1c17095522c4defb17da62f3338b0b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1c17095522c4defb17da62f3338b0b8" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39963 } health_report { overall_health: HEALTHY } } }
I20260926 12:38:30.961603 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:30.958063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.10.224.131:39963)
0926 12:38:30.958417 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:30.958424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:30.958438 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:38:30.958481 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:30.958483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:30.958536 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:30.958616 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:30.958869 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.959509 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.960368 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:30.960704 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:30.960835 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:30.961122 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:30.961216 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:30.961315 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:30.961396 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 12:38:30.961454 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":142,"threads_started":1}
I20260926 12:38:30.969151 13168 raft_consensus.cc:1261] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Refusing update from remote peer b1c17095522c4defb17da62f3338b0b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:30.969620 13028 raft_consensus.cc:1261] T d0abe0f72b9a4c29b183145aadbda1ee P 1034bdda96f24a99a0a95649b7632c2f [term 1 FOLLOWER]: Refusing update from remote peer b1c17095522c4defb17da62f3338b0b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:30.969619 13482 consensus_queue.cc:1035] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d55f9c919de44a9783165a90d2215c3f" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:30.970039 13482 consensus_queue.cc:1035] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1034bdda96f24a99a0a95649b7632c2f" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:38:30.983261 13447 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 12:38:32.540535 12871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:32.540688 12871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:32.543246 12871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7591d95c7d374de5b4529ff7fd57afda: Sending DeleteTablet for 3 replicas of tablet d0abe0f72b9a4c29b183145aadbda1ee
I20260926 12:38:32.544065 13009 tablet_service.cc:1514] Processing DeleteTablet for tablet d0abe0f72b9a4c29b183145aadbda1ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:32 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:60346
I20260926 12:38:32.544149 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet d0abe0f72b9a4c29b183145aadbda1ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:32 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:35254
I20260926 12:38:32.544158 13148 tablet_service.cc:1514] Processing DeleteTablet for tablet d0abe0f72b9a4c29b183145aadbda1ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:32 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:53720
I20260926 12:38:32.544559 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 12944
I20260926 12:38:32.547773 13526 tablet_replica.cc:330] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: stopping tablet replica
I20260926 12:38:32.547782 13527 tablet_replica.cc:330] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: stopping tablet replica
I20260926 12:38:32.547961 13526 raft_consensus.cc:2229] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:32.547987 13527 raft_consensus.cc:2229] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:32.548146 13527 raft_consensus.cc:2258] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:32.548249 13526 raft_consensus.cc:2258] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:32.549009 13526 ts_tablet_manager.cc:1905] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:32.549009 13527 ts_tablet_manager.cc:1905] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:32.552392 13526 ts_tablet_manager.cc:1918] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:38:32.552392 13527 ts_tablet_manager.cc:1918] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:38:32.552487 13526 log.cc:1196] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/d0abe0f72b9a4c29b183145aadbda1ee
I20260926 12:38:32.552487 13527 log.cc:1196] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/d0abe0f72b9a4c29b183145aadbda1ee
I20260926 12:38:32.552762 13526 ts_tablet_manager.cc:1939] T d0abe0f72b9a4c29b183145aadbda1ee P b1c17095522c4defb17da62f3338b0b8: Deleting consensus metadata
I20260926 12:38:32.552762 13527 ts_tablet_manager.cc:1939] T d0abe0f72b9a4c29b183145aadbda1ee P d55f9c919de44a9783165a90d2215c3f: Deleting consensus metadata
I20260926 12:38:32.553556 12857 catalog_manager.cc:4841] TS d55f9c919de44a9783165a90d2215c3f (127.10.224.130:38403): tablet d0abe0f72b9a4c29b183145aadbda1ee (table test-table [id=a2c104b5e0b34c2e95a106fcd60c57b6]) successfully deleted
I20260926 12:38:32.553799 12857 catalog_manager.cc:4841] TS b1c17095522c4defb17da62f3338b0b8 (127.10.224.131:39963): tablet d0abe0f72b9a4c29b183145aadbda1ee (table test-table [id=a2c104b5e0b34c2e95a106fcd60c57b6]) successfully deleted
W20260926 12:38:32.554252 12859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:40987 (error 108)
I20260926 12:38:32.554544 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 13084
I20260926 12:38:32.556209 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:32.555741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40987 (local address 127.0.0.1:60376)
0926 12:38:32.556032 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:32.556115 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:40987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":131,"threads_started":1}
W20260926 12:38:32.556485 12859 catalog_manager.cc:4584] TS 1034bdda96f24a99a0a95649b7632c2f (127.10.224.129:40987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0abe0f72b9a4c29b183145aadbda1ee: Network error: Client connection negotiation failed: client connection to 127.10.224.129:40987: connect: Connection refused (error 111)
I20260926 12:38:32.563035 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 13224
I20260926 12:38:32.570297 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 12840
2026-09-26T12:38:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6807 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:32 dist-test-slave-m3pj krb5kdc[13532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:32 dist-test-slave-m3pj krb5kdc[13532](info): set up 2 sockets
Sep 26 12:38:32 dist-test-slave-m3pj krb5kdc[13532](info): commencing operation
krb5kdc: starting...
W20260926 12:38:33.660126 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:33 dist-test-slave-m3pj krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:33.790690 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:32991
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:32991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:33.906102 13545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:33.906409 13545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:33.906452 13545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:33.910657 13545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:33.910744 13545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:33.910764 13545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:33.910782 13545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:33.910804 13545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:33.916555 13545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:32991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:33.917821 13545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:33.918952 13545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:33.928785 13550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:33.928795 13551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:33.928893 13545 server_base.cc:1034] running on GCE node
W20260926 12:38:33.928795 13553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:33.929734 13545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:33.930912 13545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:33.932093 13545 hybrid_clock.cc:648] HybridClock initialized: now 1790426313932052 us; error 52 us; skew 500 ppm
Sep 26 12:38:33 dist-test-slave-m3pj krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426313, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:33.935066 13545 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:33.936132 13545 webserver.cc:460] Webserver started at http://127.10.224.190:39087/ using document root <none> and password file <none>
I20260926 12:38:33.936698 13545 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:33.936760 13545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:33.936950 13545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:33.939177 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "e9dbe8c0b32349af84b453386f5a2f25"
format_stamp: "Formatted at 2026-09-26 12:38:33 on dist-test-slave-m3pj"
server_key: "1568744425d9e6fe1d8a3e78590ff301"
server_key_iv: "a7ffc0c6084315bd2e9d377b1512ad50"
server_key_version: "encryptionkey@0"
I20260926 12:38:33.939837 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "e9dbe8c0b32349af84b453386f5a2f25"
format_stamp: "Formatted at 2026-09-26 12:38:33 on dist-test-slave-m3pj"
server_key: "1568744425d9e6fe1d8a3e78590ff301"
server_key_iv: "a7ffc0c6084315bd2e9d377b1512ad50"
server_key_version: "encryptionkey@0"
I20260926 12:38:33.943969 13545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:38:33.946668 13560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:33.948323 13545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:33.948472 13545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "e9dbe8c0b32349af84b453386f5a2f25"
format_stamp: "Formatted at 2026-09-26 12:38:33 on dist-test-slave-m3pj"
server_key: "1568744425d9e6fe1d8a3e78590ff301"
server_key_iv: "a7ffc0c6084315bd2e9d377b1512ad50"
server_key_version: "encryptionkey@0"
I20260926 12:38:33.948598 13545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:33.962985 13545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:33.966552 13545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:33.966777 13545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:33.974185 13545 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:32991
I20260926 12:38:33.974208 13612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:32991 every 8 connection(s)
I20260926 12:38:33.975257 13545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:33.978138 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 13545
I20260926 12:38:33.978260 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:33.978500 13613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:33.978554 11138 external_mini_cluster.cc:1428] Setting key 3f425e6e0ff3ccd437a014527325d92b
I20260926 12:38:33.985594 13613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25: Bootstrap starting.
Sep 26 12:38:33 dist-test-slave-m3pj krb5kdc[13532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:33.988358 13613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:33.989226 13613 log.cc:824] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:33.991983 13613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25: No bootstrap required, opened a new log
I20260926 12:38:33.994014 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:33.980210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.10.224.190:32991)
0926 12:38:33.980679 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:33.980694 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:33.980731 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:38:33.981566 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:33.981588 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:33.981956 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:33.982310 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:33.983201 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:33.984331 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:33.985213 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:33.985773 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:33.985949 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:33.988643 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:33.988649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:33.988666 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:33.988698 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:33.990786 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:33.991336 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:33.991341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:33.991346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:33.991506 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:33.991908 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:33.991910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:33.991911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:33.992404 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:33.992564 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:33.992863 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 12:38:33.993172 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":133,"threads_started":1}
I20260926 12:38:33.996191 13613 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dbe8c0b32349af84b453386f5a2f25" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 32991 } }
I20260926 12:38:33.996524 13613 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:33.996587 13613 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9dbe8c0b32349af84b453386f5a2f25, State: Initialized, Role: FOLLOWER
I20260926 12:38:33.997056 13613 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dbe8c0b32349af84b453386f5a2f25" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 32991 } }
I20260926 12:38:33.997212 13613 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:33.997303 13613 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:33.997430 13613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:33.998436 13613 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dbe8c0b32349af84b453386f5a2f25" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 32991 } }
I20260926 12:38:33.998842 13613 leader_election.cc:302] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9dbe8c0b32349af84b453386f5a2f25; no voters:
I20260926 12:38:33.999186 13613 leader_election.cc:288] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:33.999365 13618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:33.999673 13618 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [term 1 LEADER]: Becoming Leader. State: Replica: e9dbe8c0b32349af84b453386f5a2f25, State: Running, Role: LEADER
I20260926 12:38:34.000020 13618 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dbe8c0b32349af84b453386f5a2f25" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 32991 } }
I20260926 12:38:34.000550 13613 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:34.001827 13619 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9dbe8c0b32349af84b453386f5a2f25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dbe8c0b32349af84b453386f5a2f25" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 32991 } } }
I20260926 12:38:34.001981 13619 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:34.002307 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9dbe8c0b32349af84b453386f5a2f25. Latest consensus state: current_term: 1 leader_uuid: "e9dbe8c0b32349af84b453386f5a2f25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dbe8c0b32349af84b453386f5a2f25" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 32991 } } }
I20260926 12:38:34.002405 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25 [sys.catalog]: This master's current role is: LEADER
W20260926 12:38:34.005669 13633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:34.005748 13633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:34.005834 13627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:34.006621 13627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:34.013446 13627 catalog_manager.cc:1347] Generated new cluster ID: 4827e5f0fc484963848e9d8762f70372
I20260926 12:38:34.013554 13627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:34.028935 13627 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:34.030349 13627 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:34.037951 13627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9dbe8c0b32349af84b453386f5a2f25: Generated new TSK 0
I20260926 12:38:34.038892 13627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:34.072515 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:32991
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:34.190398 13639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:34.190701 13639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:34.190747 13639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:34.195027 13639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:34.195101 13639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:34.195165 13639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:34.200996 13639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:34.202287 13639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:34.203414 13639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:34.213354 13645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:34.213455 13644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:34.213351 13647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:34.213789 13639 server_base.cc:1034] running on GCE node
I20260926 12:38:34.214253 13639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:34.215060 13639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:34.216271 13639 hybrid_clock.cc:648] HybridClock initialized: now 1790426314216214 us; error 81 us; skew 500 ppm
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:34.219748 13639 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:34.220697 13639 webserver.cc:460] Webserver started at http://127.10.224.129:37055/ using document root <none> and password file <none>
I20260926 12:38:34.221381 13639 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:34.221484 13639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:34.221724 13639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:34.223901 13639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "8fb4e50856514b219b3145a03a5b36d1"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "ce76590eccef32d7c462fa2f449c5a16"
server_key_iv: "790b9d8385d4c337f020f048eaff9a9e"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.224483 13639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "8fb4e50856514b219b3145a03a5b36d1"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "ce76590eccef32d7c462fa2f449c5a16"
server_key_iv: "790b9d8385d4c337f020f048eaff9a9e"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.229305 13639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 12:38:34.231951 13654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.233268 13639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:38:34.233431 13639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "8fb4e50856514b219b3145a03a5b36d1"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "ce76590eccef32d7c462fa2f449c5a16"
server_key_iv: "790b9d8385d4c337f020f048eaff9a9e"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.233558 13639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:34.257125 13639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:34.260763 13639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:34.260980 13639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:34.261631 13639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:34.262491 13639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:34.262562 13639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.262634 13639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:34.262686 13639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.272231 13639 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:38647
I20260926 12:38:34.272251 13767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:38647 every 8 connection(s)
I20260926 12:38:34.273221 13639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:34.280169 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 13639
I20260926 12:38:34.280274 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:34.280599 11138 external_mini_cluster.cc:1428] Setting key e45c7324e6c518fdee48d0056eb6703c
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[13532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:34.288328 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:34.275154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:39707 (local address 127.10.224.190:32991)
0926 12:38:34.275310 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:34.275319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:34.276711 (+ 1392us) server_negotiation.cc:407] Connection header received
0926 12:38:34.277654 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:34.277657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:34.277705 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:34.277782 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:34.279059 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.279645 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.281086 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.281417 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.281524 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:34.284521 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:34.284524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:34.284526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:34.284552 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:34.286146 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:34.286679 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:34.286683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:34.286685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:34.286769 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:34.287118 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:34.287123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:34.287124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:34.287317 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:34.287426 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:34.288031 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 12:38:34.288160 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 12:38:34.289196 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:34.275588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:32991 (local address 127.10.224.129:39707)
0926 12:38:34.276536 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:34.276584 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:34.277444 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:34.277974 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:34.277982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:34.278376 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:34.278913 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:34.278928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.279770 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.279773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:34.280962 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:34.280973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.281570 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.281573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:34.281738 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:34.282609 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:34.282628 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:34.284330 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:34.286268 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:34.286274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:34.286292 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:34.286573 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:34.286867 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:34.286870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:34.286872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:34.287021 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:34.287428 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:34.287437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:34.287780 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 12:38:34.287980 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 12:38:34.288332 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":542,"threads_started":1}
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:38:34.290593 13768 heartbeater.cc:347] Connected to a master server at 127.10.224.190:32991
I20260926 12:38:34.290890 13768 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:34.291574 13768 heartbeater.cc:511] Master 127.10.224.190:32991 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:38:34.293273 13577 ts_manager.cc:194] Registered new tserver with Master: 8fb4e50856514b219b3145a03a5b36d1 (127.10.224.129:38647)
I20260926 12:38:34.294957 13577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:39707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:34.311359 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:32991
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:34.428045 13773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:34.428375 13773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:34.428472 13773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:34.433074 13773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:34.433176 13773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:34.433275 13773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:34.439072 13773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:34.440368 13773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:34.441504 13773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:34.451815 13778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:34.451874 13773 server_base.cc:1034] running on GCE node
W20260926 12:38:34.451771 13781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:34.451797 13779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:34.452497 13773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:34.453260 13773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:34.454479 13773 hybrid_clock.cc:648] HybridClock initialized: now 1790426314454456 us; error 39 us; skew 500 ppm
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:34.462989 13773 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:34.464167 13773 webserver.cc:460] Webserver started at http://127.10.224.130:46229/ using document root <none> and password file <none>
I20260926 12:38:34.464857 13773 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:34.464958 13773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:34.465214 13773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:34.467406 13773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "1579865136d64e3fb52c38bd5b122fe5"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "e6539b45a9982f1bae8ac1f88eb7b7ed"
server_key_iv: "c3460ec02a2289d59c3607ec809ec957"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.468123 13773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "1579865136d64e3fb52c38bd5b122fe5"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "e6539b45a9982f1bae8ac1f88eb7b7ed"
server_key_iv: "c3460ec02a2289d59c3607ec809ec957"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.472294 13773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:34.475257 13788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.477427 13773 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:38:34.477610 13773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "1579865136d64e3fb52c38bd5b122fe5"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "e6539b45a9982f1bae8ac1f88eb7b7ed"
server_key_iv: "c3460ec02a2289d59c3607ec809ec957"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.477778 13773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:34.507265 13773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:34.511965 13773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:34.512223 13773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:34.512979 13773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:34.513983 13773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:34.514035 13773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.514127 13773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:34.514163 13773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.525238 13773 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:32775
I20260926 12:38:34.525257 13901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:32775 every 8 connection(s)
I20260926 12:38:34.526317 13773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:34.530514 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 13773
I20260926 12:38:34.530611 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:34.530886 11138 external_mini_cluster.cc:1428] Setting key cc79b16f83b2053184a0ebd2a49d9dc7
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[13532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:38:34.543458 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:34.528874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:47759 (local address 127.10.224.190:32991)
0926 12:38:34.529020 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:34.529024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:34.530184 (+ 1160us) server_negotiation.cc:407] Connection header received
0926 12:38:34.531249 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:34.531264 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:34.531328 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:34.531407 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:34.532814 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.533684 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.535500 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.535981 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.536114 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:34.539407 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:34.539413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:34.539416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:34.539452 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:34.541245 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:34.541848 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:34.541852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:34.541854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:34.541947 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:34.542359 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:34.542363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:34.542365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:34.542564 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:34.542646 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:34.543206 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 12:38:34.543325 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:38:34.544569 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:34.529337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:32991 (local address 127.10.224.130:47759)
0926 12:38:34.530025 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:34.530069 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:34.531022 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:34.531691 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:34.531700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:34.532065 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:34.532670 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:34.532685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.533811 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.533814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:34.535366 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:34.535376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.536143 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.536146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:34.536294 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:34.537067 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:34.537097 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:34.539211 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:34.541406 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:34.541411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:34.541435 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:34.541733 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:34.542072 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:34.542075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:34.542079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:34.542255 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:34.542664 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:34.542670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:34.542958 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 12:38:34.543175 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 12:38:34.543581 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:38:34.545856 13902 heartbeater.cc:347] Connected to a master server at 127.10.224.190:32991
I20260926 12:38:34.546131 13902 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:34.546770 13902 heartbeater.cc:511] Master 127.10.224.190:32991 requested a full tablet report, sending...
I20260926 12:38:34.548269 13577 ts_manager.cc:194] Registered new tserver with Master: 1579865136d64e3fb52c38bd5b122fe5 (127.10.224.130:32775)
I20260926 12:38:34.548904 13577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:47759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:34.563575 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:32991
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:34.680332 13907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:34.680619 13907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:34.680663 13907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:34.684844 13907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:34.684911 13907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:34.685012 13907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:34.690541 13907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:34.691622 13907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:34.692622 13907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:34.703018 13915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:34.703078 13913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:34.703076 13912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:34.703560 13907 server_base.cc:1034] running on GCE node
I20260926 12:38:34.704057 13907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:34.704768 13907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:34.705972 13907 hybrid_clock.cc:648] HybridClock initialized: now 1790426314705955 us; error 38 us; skew 500 ppm
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:34.708983 13907 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:34.710040 13907 webserver.cc:460] Webserver started at http://127.10.224.131:32985/ using document root <none> and password file <none>
I20260926 12:38:34.710659 13907 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:34.710754 13907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:34.711004 13907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:34.713259 13907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "1c1624f6209749419a59f1d445afaf0d"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "ef9005ce34fe5411dc178aa0da997fe4"
server_key_iv: "aec48d0b583261cea009cb77b7e4006b"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.713871 13907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "1c1624f6209749419a59f1d445afaf0d"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "ef9005ce34fe5411dc178aa0da997fe4"
server_key_iv: "aec48d0b583261cea009cb77b7e4006b"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.717944 13907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:34.720624 13922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.721982 13907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:34.722134 13907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "1c1624f6209749419a59f1d445afaf0d"
format_stamp: "Formatted at 2026-09-26 12:38:34 on dist-test-slave-m3pj"
server_key: "ef9005ce34fe5411dc178aa0da997fe4"
server_key_iv: "aec48d0b583261cea009cb77b7e4006b"
server_key_version: "encryptionkey@0"
I20260926 12:38:34.722265 13907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:34.743084 13907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:34.746469 13907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:34.746702 13907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:34.747361 13907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:34.748397 13907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:34.748481 13907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.748556 13907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:34.748607 13907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:34.758165 13907 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:43015
I20260926 12:38:34.758188 14035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:43015 every 8 connection(s)
I20260926 12:38:34.759120 13907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:34.760471 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 13907
I20260926 12:38:34.760596 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:34.760911 11138 external_mini_cluster.cc:1428] Setting key c5ba2fe41ed47e3bf63da08af0b355ce
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[13532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426314, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:34.773684 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:34.761424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:53145 (local address 127.10.224.190:32991)
0926 12:38:34.761577 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:34.761581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:34.762843 (+ 1262us) server_negotiation.cc:407] Connection header received
0926 12:38:34.763834 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:34.763837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:34.763885 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:34.763960 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:34.765279 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.765850 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.767033 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.767254 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.767338 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:34.769887 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:34.769891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:34.769894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:34.769921 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:34.771434 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:34.771973 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:34.771975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:34.771977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:34.772053 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:34.772358 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:34.772360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:34.772361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:34.772580 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:34.772659 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:34.773362 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 12:38:34.773485 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:38:34.774690 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:34.761933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:32991 (local address 127.10.224.131:53145)
0926 12:38:34.762680 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:34.762734 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:34.763631 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:34.764139 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:34.764146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:34.764612 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:34.765124 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:34.765138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.765980 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.765983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:34.766908 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:34.766916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:34.767395 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:34.767397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:34.767617 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:34.768175 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:34.768191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:34.769721 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:34.771607 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:34.771612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:34.771626 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:34.771889 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:34.772134 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:34.772136 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:34.772138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:34.772269 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:34.772710 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:34.772716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:34.773012 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 12:38:34.773199 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 12:38:34.773603 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":146,"threads_started":1}
I20260926 12:38:34.775976 14036 heartbeater.cc:347] Connected to a master server at 127.10.224.190:32991
I20260926 12:38:34.776243 14036 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:34.776823 14036 heartbeater.cc:511] Master 127.10.224.190:32991 requested a full tablet report, sending...
I20260926 12:38:34.777935 13577 ts_manager.cc:194] Registered new tserver with Master: 1c1624f6209749419a59f1d445afaf0d (127.10.224.131:43015)
I20260926 12:38:34.778548 13577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:53145
I20260926 12:38:34.787720 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:38:34.802501 13616 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:34.800648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.10.224.190:32991)
0926 12:38:34.800822 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:34.800828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:34.800963 (+ 135us) server_negotiation.cc:407] Connection header received
0926 12:38:34.801166 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:34.801169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:34.801215 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:34.801319 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:34.802324 (+ 1005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55326: BlockingRecv error: recv got EOF from 127.0.0.1:55326 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:38:34.803848 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 13639
I20260926 12:38:34.809278 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 13773
I20260926 12:38:34.814992 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 13907
I20260926 12:38:34.821617 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 13545
2026-09-26T12:38:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2249 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[14048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[14048](info): set up 2 sockets
Sep 26 12:38:34 dist-test-slave-m3pj krb5kdc[14048](info): commencing operation
krb5kdc: starting...
W20260926 12:38:35.896620 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 12:38:36.179370 11138 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.201s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/test-user.keytab.
W20260926 12:38:37.024711 11138 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.752s user 0.002s sys 0.000s
Sep 26 12:38:37 dist-test-slave-m3pj krb5kdc[14048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
W20260926 12:38:38.256722 11138 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.224.190: real 1.207s user 0.000s sys 0.004s
I20260926 12:38:38.258534 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:43443
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:43443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:38.395944 14061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:38.396358 14061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:38.396409 14061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:38.401134 14061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:38.401244 14061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:38.401266 14061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:38.401304 14061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:38.401320 14061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:38.407706 14061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:43443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:38.409343 14061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:38.410821 14061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:38.421413 14067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:38.421413 14069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:38.421470 14066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:38.421799 14061 server_base.cc:1034] running on GCE node
I20260926 12:38:38.422542 14061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:38.424285 14061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:38.425513 14061 hybrid_clock.cc:648] HybridClock initialized: now 1790426318425484 us; error 52 us; skew 500 ppm
Sep 26 12:38:38 dist-test-slave-m3pj krb5kdc[14048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426318, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:38.437738 14061 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:38.439491 14061 webserver.cc:460] Webserver started at http://127.10.224.190:33039/ using document root <none> and password file <none>
I20260926 12:38:38.440433 14061 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:38.440557 14061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:38.441797 14061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:38.445036 14061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "7938de5dc93f4aa898373569ae32c09e"
format_stamp: "Formatted at 2026-09-26 12:38:38 on dist-test-slave-m3pj"
server_key: "fe788e6dd645736d0141ba2f2720ce41"
server_key_iv: "70dd5dfd391b24f9d4dc1a97c99fc6df"
server_key_version: "encryptionkey@0"
I20260926 12:38:38.446070 14061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "7938de5dc93f4aa898373569ae32c09e"
format_stamp: "Formatted at 2026-09-26 12:38:38 on dist-test-slave-m3pj"
server_key: "fe788e6dd645736d0141ba2f2720ce41"
server_key_iv: "70dd5dfd391b24f9d4dc1a97c99fc6df"
server_key_version: "encryptionkey@0"
I20260926 12:38:38.451686 14061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 12:38:38.455056 14076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:38.456907 14061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:38:38.457074 14061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "7938de5dc93f4aa898373569ae32c09e"
format_stamp: "Formatted at 2026-09-26 12:38:38 on dist-test-slave-m3pj"
server_key: "fe788e6dd645736d0141ba2f2720ce41"
server_key_iv: "70dd5dfd391b24f9d4dc1a97c99fc6df"
server_key_version: "encryptionkey@0"
I20260926 12:38:38.457227 14061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:38.472626 14061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:38.476892 14061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:38.477164 14061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:38.486567 14061 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:43443
I20260926 12:38:38.486914 14128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:43443 every 8 connection(s)
I20260926 12:38:38.487954 14061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:38.492096 14129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:38.498095 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 14061
I20260926 12:38:38.498255 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:38.498562 11138 external_mini_cluster.cc:1428] Setting key d452a447fc6f59472b6b90050d0ae46b
I20260926 12:38:38.500028 14129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e: Bootstrap starting.
I20260926 12:38:38.503327 14129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:38.504334 14129 log.cc:824] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:38.506892 14129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e: No bootstrap required, opened a new log
Sep 26 12:38:38 dist-test-slave-m3pj krb5kdc[14048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:38.511837 14129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7938de5dc93f4aa898373569ae32c09e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43443 } }
I20260926 12:38:38.512247 14129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:38.512338 14129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7938de5dc93f4aa898373569ae32c09e, State: Initialized, Role: FOLLOWER
I20260926 12:38:38.513072 14129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7938de5dc93f4aa898373569ae32c09e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43443 } }
I20260926 12:38:38.513279 14129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:38.513361 14129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:38.513475 14129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:38.516167 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:38.500487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.10.224.190:43443)
0926 12:38:38.501069 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:38.501089 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:38.501134 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:38:38.501930 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:38.501953 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:38.502394 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:38.502884 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:38.503966 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:38.505113 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:38.506073 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:38.506611 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:38.506810 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:38.509664 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:38.509671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:38.509690 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:38.509724 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:38.511934 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:38.512538 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:38.512546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:38.512554 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:38.512738 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:38.513083 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:38.513090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:38.513094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:38.513848 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:38.514063 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:38.514399 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 12:38:38.514832 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":193,"threads_started":1}
I20260926 12:38:38.520007 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:38:38.540123 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:38.534532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.10.224.190:43443)
0926 12:38:38.534698 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:38.534703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:38.534932 (+ 229us) server_negotiation.cc:407] Connection header received
0926 12:38:38.535028 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:38.535031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:38.535085 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:38.535189 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:38.535881 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:38.536468 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:38.537414 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:38.537696 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:38.537790 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:38.538735 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:38.538739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:38.538741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:38.539634 (+ 893us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:38.539673 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:38.539695 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:38.539788 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:38.539917 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 12:38:38.539984 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:38:38.556685 14129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7938de5dc93f4aa898373569ae32c09e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43443 } }
I20260926 12:38:38.557381 14129 leader_election.cc:302] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7938de5dc93f4aa898373569ae32c09e; no voters:
I20260926 12:38:38.557960 14129 leader_election.cc:288] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:38.558094 14141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:38.558440 14141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [term 1 LEADER]: Becoming Leader. State: Replica: 7938de5dc93f4aa898373569ae32c09e, State: Running, Role: LEADER
I20260926 12:38:38.558938 14141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7938de5dc93f4aa898373569ae32c09e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43443 } }
I20260926 12:38:38.560180 14129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:38.560969 14143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7938de5dc93f4aa898373569ae32c09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7938de5dc93f4aa898373569ae32c09e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43443 } } }
I20260926 12:38:38.561125 14143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:38.561671 14142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7938de5dc93f4aa898373569ae32c09e. Latest consensus state: current_term: 1 leader_uuid: "7938de5dc93f4aa898373569ae32c09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7938de5dc93f4aa898373569ae32c09e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43443 } } }
I20260926 12:38:38.561841 14142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:38.563112 14147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:38.566522 14147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:38.573012 14147 catalog_manager.cc:1347] Generated new cluster ID: 47c177fa81c544208bb9c51811051b3d
I20260926 12:38:38.573127 14147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:38.591113 14147 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:38.592289 14147 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:38.597707 14147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7938de5dc93f4aa898373569ae32c09e: Generated new TSK 0
I20260926 12:38:38.598513 14147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:38:38.616264 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 14061
2026-09-26T12:38:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3791 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:40 dist-test-slave-m3pj krb5kdc[14162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:40 dist-test-slave-m3pj krb5kdc[14162](info): set up 2 sockets
Sep 26 12:38:40 dist-test-slave-m3pj krb5kdc[14162](info): commencing operation
krb5kdc: starting...
W20260926 12:38:40.980139 11138 mini_kdc.cc:121] Time spent starting KDC: real 2.343s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:41 dist-test-slave-m3pj krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:41.115480 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:40587
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:41.240667 14175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:41.241048 14175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:41.241127 14175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:41.245708 14175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:41.245836 14175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:41.245874 14175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:41.245918 14175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:41.245957 14175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:41.253096 14175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:40587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:41.254803 14175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:41.256505 14175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:41.269742 14181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:41.269640 14180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:41.270092 14175 server_base.cc:1034] running on GCE node
W20260926 12:38:41.269658 14183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:41.271099 14175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:41.272761 14175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:41.274046 14175 hybrid_clock.cc:648] HybridClock initialized: now 1790426321274024 us; error 40 us; skew 500 ppm
Sep 26 12:38:41 dist-test-slave-m3pj krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:41.277684 14175 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:41.278903 14175 webserver.cc:460] Webserver started at http://127.10.224.190:34059/ using document root <none> and password file <none>
I20260926 12:38:41.279665 14175 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:41.279744 14175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:41.279954 14175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:41.282538 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "ae6d60b79d554820937b0e51dceb3657"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "c91195b10d794136f31ae03344bf5ab7"
server_key_iv: "85d2d26cc33cc1c27e12fae97fb6510f"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.283125 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "ae6d60b79d554820937b0e51dceb3657"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "c91195b10d794136f31ae03344bf5ab7"
server_key_iv: "85d2d26cc33cc1c27e12fae97fb6510f"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.287660 14175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 12:38:41.290757 14190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:41.292676 14175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:41.292862 14175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "ae6d60b79d554820937b0e51dceb3657"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "c91195b10d794136f31ae03344bf5ab7"
server_key_iv: "85d2d26cc33cc1c27e12fae97fb6510f"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.293002 14175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:41.317477 14175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:41.321475 14175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:41.321707 14175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:41.329433 14175 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:40587
I20260926 12:38:41.329734 14242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:40587 every 8 connection(s)
I20260926 12:38:41.330579 14175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:41.333398 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 14175
I20260926 12:38:41.333572 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:41.333907 11138 external_mini_cluster.cc:1428] Setting key e33bbf9b27536b1cd930ca196e95709d
I20260926 12:38:41.333968 14243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:41.341082 14243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657: Bootstrap starting.
Sep 26 12:38:41 dist-test-slave-m3pj krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:41.344066 14243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:41.344921 14243 log.cc:824] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:41.347306 14243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657: No bootstrap required, opened a new log
I20260926 12:38:41.349650 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:41.335560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.10.224.190:40587)
0926 12:38:41.336143 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:41.336160 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:41.336201 (+ 41us) server_negotiation.cc:407] Connection header received
0926 12:38:41.336988 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:41.337008 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:41.337456 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:41.337832 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:41.338716 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.339864 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.340766 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.341350 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.341530 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:41.344529 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:41.344535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:41.344551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:41.344583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:41.346503 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:41.347079 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:41.347085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:41.347090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:41.347207 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:41.347558 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:41.347561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:41.347563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:41.348009 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:41.348228 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:41.348508 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 12:38:41.348853 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":161,"threads_started":1}
I20260926 12:38:41.351877 14243 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6d60b79d554820937b0e51dceb3657" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 40587 } }
I20260926 12:38:41.352203 14243 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:41.352299 14243 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6d60b79d554820937b0e51dceb3657, State: Initialized, Role: FOLLOWER
I20260926 12:38:41.352778 14243 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6d60b79d554820937b0e51dceb3657" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 40587 } }
I20260926 12:38:41.352931 14243 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:41.353019 14243 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:41.353147 14243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:41.354185 14243 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6d60b79d554820937b0e51dceb3657" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 40587 } }
I20260926 12:38:41.354566 14243 leader_election.cc:302] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6d60b79d554820937b0e51dceb3657; no voters:
I20260926 12:38:41.354908 14243 leader_election.cc:288] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:41.355101 14248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:41.355363 14248 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [term 1 LEADER]: Becoming Leader. State: Replica: ae6d60b79d554820937b0e51dceb3657, State: Running, Role: LEADER
I20260926 12:38:41.355736 14248 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6d60b79d554820937b0e51dceb3657" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 40587 } }
I20260926 12:38:41.356333 14243 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:41.357627 14250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6d60b79d554820937b0e51dceb3657. Latest consensus state: current_term: 1 leader_uuid: "ae6d60b79d554820937b0e51dceb3657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6d60b79d554820937b0e51dceb3657" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 40587 } } }
I20260926 12:38:41.357790 14250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:41.358160 14249 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6d60b79d554820937b0e51dceb3657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6d60b79d554820937b0e51dceb3657" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 40587 } } }
I20260926 12:38:41.358247 14249 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657 [sys.catalog]: This master's current role is: LEADER
W20260926 12:38:41.361629 14263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:41.361748 14263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:41.361827 14257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:41.362924 14257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:41.369205 14257 catalog_manager.cc:1347] Generated new cluster ID: 4c62ed9bd5f3455d85980a4e2464ffb3
I20260926 12:38:41.369297 14257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:41.399016 14257 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:41.400137 14257 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:41.408489 14257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6d60b79d554820937b0e51dceb3657: Generated new TSK 0
I20260926 12:38:41.409298 14257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:41.447127 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:40587
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:41.566815 14269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:41.567139 14269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:41.567186 14269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:41.571382 14269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:41.571471 14269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:41.571557 14269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:41.577627 14269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:40587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:41.578805 14269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:41.579887 14269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:41.590305 14277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:41.590303 14274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:41.590304 14275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:41.590366 14269 server_base.cc:1034] running on GCE node
I20260926 12:38:41.591142 14269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:41.591879 14269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:41.593042 14269 hybrid_clock.cc:648] HybridClock initialized: now 1790426321593025 us; error 31 us; skew 500 ppm
Sep 26 12:38:41 dist-test-slave-m3pj krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:41.596091 14269 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:41.597204 14269 webserver.cc:460] Webserver started at http://127.10.224.129:42259/ using document root <none> and password file <none>
I20260926 12:38:41.597781 14269 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:41.597846 14269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:41.598034 14269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:41.600401 14269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "809d823734e24737a4e1f38cec63a03c"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "11087beab03e19113d118c2b9d98644c"
server_key_iv: "2337273dc10bd2a80d07d472673fa02b"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.601020 14269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "809d823734e24737a4e1f38cec63a03c"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "11087beab03e19113d118c2b9d98644c"
server_key_iv: "2337273dc10bd2a80d07d472673fa02b"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.605237 14269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:41.608134 14284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:41.609516 14269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:41.609656 14269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "809d823734e24737a4e1f38cec63a03c"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "11087beab03e19113d118c2b9d98644c"
server_key_iv: "2337273dc10bd2a80d07d472673fa02b"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.609777 14269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:41.626914 14269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:41.630157 14269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:41.630378 14269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:41.631033 14269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:41.632000 14269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:41.632050 14269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:41.632123 14269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:41.632154 14269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:41.641343 14269 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:41805
I20260926 12:38:41.641360 14397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:41805 every 8 connection(s)
I20260926 12:38:41.642400 14269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:41.644767 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 14269
I20260926 12:38:41.644918 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:41.645234 11138 external_mini_cluster.cc:1428] Setting key 3b2251c09a14333b173ba601b7b24e66
Sep 26 12:38:41 dist-test-slave-m3pj krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:38:41.659557 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:41.644584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:51071 (local address 127.10.224.190:40587)
0926 12:38:41.644801 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:41.644806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:41.645878 (+ 1072us) server_negotiation.cc:407] Connection header received
0926 12:38:41.647412 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:41.647415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:41.647541 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:41.647615 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:41.648893 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.649440 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.651347 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.651679 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.651774 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:41.655046 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:41.655050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:41.655053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:41.655089 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:41.656759 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:41.657439 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:41.657448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:41.657449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:41.657529 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:41.657909 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:41.657914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:41.657915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:41.658250 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:41.658358 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:41.659235 (+ 877us) server_negotiation.cc:299] Negotiation successful
0926 12:38:41.659372 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 12:38:41.660918 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:41.645126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:40587 (local address 127.10.224.129:51071)
0926 12:38:41.645716 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:41.645774 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:41.647155 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:41.647810 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:41.647817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:41.648204 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:41.648735 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:41.648750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.649585 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.649592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:41.651206 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:41.651222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.651856 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.651864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:41.652160 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:41.652996 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:41.653025 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:41.654832 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:41.656941 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:41.656952 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:41.656975 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:41.657311 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:41.657633 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:41.657636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:41.657638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:41.657807 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:41.658373 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:41.658382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:41.658687 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 12:38:41.659133 (+ 446us) client_negotiation.cc:240] Negotiation successful
0926 12:38:41.659618 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":166,"threads_started":1}
I20260926 12:38:41.662608 14398 heartbeater.cc:347] Connected to a master server at 127.10.224.190:40587
I20260926 12:38:41.662899 14398 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:41.663465 14398 heartbeater.cc:511] Master 127.10.224.190:40587 requested a full tablet report, sending...
I20260926 12:38:41.665477 14207 ts_manager.cc:194] Registered new tserver with Master: 809d823734e24737a4e1f38cec63a03c (127.10.224.129:41805)
I20260926 12:38:41.667291 14207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:51071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:41.679666 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:40587
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:41.797410 14403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:41.797813 14403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:41.797876 14403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:41.802266 14403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:41.802357 14403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:41.802464 14403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:41.808246 14403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:40587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:41.809418 14403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:41.810500 14403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:38:41.820909 14403 server_base.cc:1034] running on GCE node
W20260926 12:38:41.820856 14409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:41.820856 14411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:41.820859 14408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:41.821578 14403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:41.822292 14403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:41.823484 14403 hybrid_clock.cc:648] HybridClock initialized: now 1790426321823425 us; error 55 us; skew 500 ppm
Sep 26 12:38:41 dist-test-slave-m3pj krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:41.826504 14403 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:41.827661 14403 webserver.cc:460] Webserver started at http://127.10.224.130:40783/ using document root <none> and password file <none>
I20260926 12:38:41.828230 14403 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:41.828297 14403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:41.828480 14403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:41.830612 14403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "4291966cf4d34944a06a2d94884c1dca"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "60ae6222192c07d628cd2f9aea5e80fb"
server_key_iv: "c767be16a929ab280686ab1f7ce98e7e"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.831148 14403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "4291966cf4d34944a06a2d94884c1dca"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "60ae6222192c07d628cd2f9aea5e80fb"
server_key_iv: "c767be16a929ab280686ab1f7ce98e7e"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.835079 14403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:41.837990 14418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:41.839404 14403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:38:41.839645 14403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "4291966cf4d34944a06a2d94884c1dca"
format_stamp: "Formatted at 2026-09-26 12:38:41 on dist-test-slave-m3pj"
server_key: "60ae6222192c07d628cd2f9aea5e80fb"
server_key_iv: "c767be16a929ab280686ab1f7ce98e7e"
server_key_version: "encryptionkey@0"
I20260926 12:38:41.839788 14403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:41.876919 14403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:41.880414 14403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:41.880653 14403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:41.881323 14403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:41.882244 14403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:41.882320 14403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:41.882395 14403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:41.882438 14403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:41.891863 14403 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:41621
I20260926 12:38:41.891912 14531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:41621 every 8 connection(s)
I20260926 12:38:41.892861 14403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:41.897424 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 14403
I20260926 12:38:41.897576 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:41.897889 11138 external_mini_cluster.cc:1428] Setting key 4a84480833062dfc02e705b0c074aad1
Sep 26 12:38:41 dist-test-slave-m3pj krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:38:41.909040 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:41.894872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:42579 (local address 127.10.224.190:40587)
0926 12:38:41.895009 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:41.895012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:41.895957 (+ 945us) server_negotiation.cc:407] Connection header received
0926 12:38:41.897485 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:41.897490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:41.897564 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:41.897666 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:41.899188 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.899861 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.901116 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.901535 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.901641 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:41.904870 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:41.904873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:41.904876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:41.904911 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:41.906574 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:41.907236 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:41.907239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:41.907241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:41.907328 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:41.907828 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:41.907831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:41.907832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:41.908034 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:41.908166 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:41.908786 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 12:38:41.908902 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:38:41.910120 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:41.895270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:40587 (local address 127.10.224.130:42579)
0926 12:38:41.895797 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:41.895841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:41.897189 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:41.897856 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:41.897864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:41.898423 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:41.899010 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:41.899027 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.900013 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.900017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:41.900933 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:41.900942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:41.901731 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:41.901739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:41.902049 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:41.902822 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:41.902840 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:41.904675 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:41.906801 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:41.906809 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:41.906828 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:41.907117 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:41.907425 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:41.907429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:41.907431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:41.907710 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:41.908165 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:41.908170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:41.908482 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 12:38:41.908716 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 12:38:41.909063 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:38:41.911386 14532 heartbeater.cc:347] Connected to a master server at 127.10.224.190:40587
I20260926 12:38:41.911710 14532 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:41.912307 14532 heartbeater.cc:511] Master 127.10.224.190:40587 requested a full tablet report, sending...
I20260926 12:38:41.913539 14207 ts_manager.cc:194] Registered new tserver with Master: 4291966cf4d34944a06a2d94884c1dca (127.10.224.130:41621)
I20260926 12:38:41.914203 14207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:42579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:41.927824 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:40587
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:42.044692 14537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:42.045037 14537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:42.045116 14537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:42.049536 14537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:42.049642 14537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:42.049741 14537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:42.055490 14537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:40587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:42.056807 14537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:42.058023 14537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:42.068468 14542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:42.068486 14545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:42.068470 14543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:42.068743 14537 server_base.cc:1034] running on GCE node
I20260926 12:38:42.069361 14537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:42.070171 14537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:42.071370 14537 hybrid_clock.cc:648] HybridClock initialized: now 1790426322071332 us; error 67 us; skew 500 ppm
Sep 26 12:38:42 dist-test-slave-m3pj krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:42.074653 14537 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:42.075878 14537 webserver.cc:460] Webserver started at http://127.10.224.131:43559/ using document root <none> and password file <none>
I20260926 12:38:42.076535 14537 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:42.076639 14537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:42.076906 14537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:42.079236 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "bb2b3f9960374b1fa3dfa2977bfee31d"
format_stamp: "Formatted at 2026-09-26 12:38:42 on dist-test-slave-m3pj"
server_key: "c85bb2b79b4a5261a292c4d7c5d79d6e"
server_key_iv: "eb64f5ebc290d4581439ea9cb75be213"
server_key_version: "encryptionkey@0"
I20260926 12:38:42.079949 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "bb2b3f9960374b1fa3dfa2977bfee31d"
format_stamp: "Formatted at 2026-09-26 12:38:42 on dist-test-slave-m3pj"
server_key: "c85bb2b79b4a5261a292c4d7c5d79d6e"
server_key_iv: "eb64f5ebc290d4581439ea9cb75be213"
server_key_version: "encryptionkey@0"
I20260926 12:38:42.084249 14537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:42.087016 14552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:42.088551 14537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:42.088716 14537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "bb2b3f9960374b1fa3dfa2977bfee31d"
format_stamp: "Formatted at 2026-09-26 12:38:42 on dist-test-slave-m3pj"
server_key: "c85bb2b79b4a5261a292c4d7c5d79d6e"
server_key_iv: "eb64f5ebc290d4581439ea9cb75be213"
server_key_version: "encryptionkey@0"
I20260926 12:38:42.088855 14537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:42.130069 14537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:42.133862 14537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:42.134096 14537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:42.134815 14537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:42.135738 14537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:42.135812 14537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:42.135886 14537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:42.135931 14537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:42.145349 14537 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:42591
I20260926 12:38:42.145387 14665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:42591 every 8 connection(s)
I20260926 12:38:42.146386 14537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:42.155154 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 14537
I20260926 12:38:42.155412 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:42.155809 11138 external_mini_cluster.cc:1428] Setting key e271989db160784b88b8eefdeffdb744
Sep 26 12:38:42 dist-test-slave-m3pj krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:42.161983 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:42.148462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:56179 (local address 127.10.224.190:40587)
0926 12:38:42.148701 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:42.148706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:42.149695 (+ 989us) server_negotiation.cc:407] Connection header received
0926 12:38:42.151060 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:42.151063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:42.151114 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:42.151194 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:42.152580 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:42.153137 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:42.154388 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:42.154639 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:42.154733 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:42.157959 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:42.157963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:42.157966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:42.157997 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:42.159741 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:42.160320 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:42.160328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:42.160330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:42.160414 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:42.160791 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:42.160796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:42.160798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:42.161000 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:42.161106 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:42.161687 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 12:38:42.161817 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 12:38:42.162887 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:42.149024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:40587 (local address 127.10.224.131:56179)
0926 12:38:42.149525 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:42.149570 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:42.150836 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:42.151382 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:42.151390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:42.151871 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:42.152410 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:42.152427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:42.153262 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:42.153265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:42.154246 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:42.154255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:42.154789 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:42.154792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:42.154942 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:42.155784 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:42.155801 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:42.157786 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:42.159879 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:42.159886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:42.159906 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:42.160202 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:42.160518 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:42.160521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:42.160523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:42.160690 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:42.161117 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:42.161125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:42.161435 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 12:38:42.161647 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 12:38:42.162032 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20260926 12:38:42.164127 14666 heartbeater.cc:347] Connected to a master server at 127.10.224.190:40587
I20260926 12:38:42.164397 14666 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:42.164955 14666 heartbeater.cc:511] Master 127.10.224.190:40587 requested a full tablet report, sending...
I20260926 12:38:42.166087 14207 ts_manager.cc:194] Registered new tserver with Master: bb2b3f9960374b1fa3dfa2977bfee31d (127.10.224.131:42591)
I20260926 12:38:42.166699 14207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:56179
I20260926 12:38:42.169921 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:38:42 dist-test-slave-m3pj krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 12:38:42 dist-test-slave-m3pj krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:42.197196 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:42.187300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.10.224.190:40587)
0926 12:38:42.187441 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:42.187446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:42.187689 (+ 243us) server_negotiation.cc:407] Connection header received
0926 12:38:42.187808 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:42.187811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:42.187856 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:42.187937 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:42.188683 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:42.189218 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:42.190126 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:42.190338 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:42.190416 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:42.193453 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:42.193457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:42.193459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:42.193489 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:42.195198 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:42.195778 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:42.195781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:42.195782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:42.195872 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:42.196140 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:42.196143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:42.196144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:42.196467 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:42.196582 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:42.196842 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 12:38:42.196985 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 12:38:42.197638 14207 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34208
I20260926 12:38:42.198685 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 14269
I20260926 12:38:42.204964 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 14403
I20260926 12:38:42.211417 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 14537
I20260926 12:38:42.217165 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 14175
2026-09-26T12:38:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3603 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:42 dist-test-slave-m3pj krb5kdc[14678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:42 dist-test-slave-m3pj krb5kdc[14678](info): set up 2 sockets
Sep 26 12:38:42 dist-test-slave-m3pj krb5kdc[14678](info): commencing operation
krb5kdc: starting...
W20260926 12:38:43.300691 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:43 dist-test-slave-m3pj krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:43.432884 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39135
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:39135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:43.546562 14691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:43.546900 14691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:43.546977 14691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:43.551162 14691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:43.551265 14691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:43.551304 14691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:43.551367 14691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:43.551406 14691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:43.557016 14691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:39135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:43.558238 14691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:43.559324 14691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:43.568218 14697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:43.568218 14696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:43.568260 14699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:43.568439 14691 server_base.cc:1034] running on GCE node
I20260926 12:38:43.569176 14691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:43.570403 14691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:43.571611 14691 hybrid_clock.cc:648] HybridClock initialized: now 1790426323571597 us; error 29 us; skew 500 ppm
Sep 26 12:38:43 dist-test-slave-m3pj krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:43.581249 14691 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:43.582453 14691 webserver.cc:460] Webserver started at http://127.10.224.190:35745/ using document root <none> and password file <none>
I20260926 12:38:43.583112 14691 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:43.583212 14691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:43.583472 14691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:43.585711 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "6c70f87a105d475ab3aa446df27c7382"
format_stamp: "Formatted at 2026-09-26 12:38:43 on dist-test-slave-m3pj"
server_key: "e5f1ff025c8b6fb0fd1d6cb53de71675"
server_key_iv: "956680d5a1276e059e14aa5c1823413d"
server_key_version: "encryptionkey@0"
I20260926 12:38:43.586318 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "6c70f87a105d475ab3aa446df27c7382"
format_stamp: "Formatted at 2026-09-26 12:38:43 on dist-test-slave-m3pj"
server_key: "e5f1ff025c8b6fb0fd1d6cb53de71675"
server_key_iv: "956680d5a1276e059e14aa5c1823413d"
server_key_version: "encryptionkey@0"
I20260926 12:38:43.590339 14691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:43.593014 14706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:43.594295 14691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:38:43.594450 14691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "6c70f87a105d475ab3aa446df27c7382"
format_stamp: "Formatted at 2026-09-26 12:38:43 on dist-test-slave-m3pj"
server_key: "e5f1ff025c8b6fb0fd1d6cb53de71675"
server_key_iv: "956680d5a1276e059e14aa5c1823413d"
server_key_version: "encryptionkey@0"
I20260926 12:38:43.594591 14691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:43.608523 14691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:43.612221 14691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:43.612450 14691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:43.619972 14691 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:39135
I20260926 12:38:43.620007 14758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:39135 every 8 connection(s)
I20260926 12:38:43.621044 14691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:43.624032 14759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:43.629786 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 14691
I20260926 12:38:43.629896 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:43.629876 14759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382: Bootstrap starting.
I20260926 12:38:43.630184 11138 external_mini_cluster.cc:1428] Setting key cfdbd52876a1459ad737469f17cd3c5f
I20260926 12:38:43.632853 14759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:43.633864 14759 log.cc:824] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:43.636304 14759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382: No bootstrap required, opened a new log
Sep 26 12:38:43 dist-test-slave-m3pj krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:43.640693 14759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c70f87a105d475ab3aa446df27c7382" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39135 } }
I20260926 12:38:43.640970 14759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:43.641048 14759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70f87a105d475ab3aa446df27c7382, State: Initialized, Role: FOLLOWER
I20260926 12:38:43.641557 14759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c70f87a105d475ab3aa446df27c7382" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39135 } }
I20260926 12:38:43.641712 14759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:43.641803 14759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:43.641907 14759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:43.642863 14759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c70f87a105d475ab3aa446df27c7382" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39135 } }
I20260926 12:38:43.643234 14759 leader_election.cc:302] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c70f87a105d475ab3aa446df27c7382; no voters:
I20260926 12:38:43.643616 14759 leader_election.cc:288] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:43.643693 14764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:43.643918 14764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [term 1 LEADER]: Becoming Leader. State: Replica: 6c70f87a105d475ab3aa446df27c7382, State: Running, Role: LEADER
I20260926 12:38:43.644307 14764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c70f87a105d475ab3aa446df27c7382" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39135 } }
I20260926 12:38:43.645033 14759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:43.646006 14765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c70f87a105d475ab3aa446df27c7382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c70f87a105d475ab3aa446df27c7382" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39135 } } }
I20260926 12:38:43.646148 14766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c70f87a105d475ab3aa446df27c7382. Latest consensus state: current_term: 1 leader_uuid: "6c70f87a105d475ab3aa446df27c7382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c70f87a105d475ab3aa446df27c7382" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39135 } } }
I20260926 12:38:43.646251 14765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:43.646365 14766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:43.646899 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:43.631781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.10.224.190:39135)
0926 12:38:43.632230 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:43.632250 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:43.632295 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:38:43.633270 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:43.633303 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:43.633786 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:43.634172 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:43.635096 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:43.636584 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:43.637431 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:43.638170 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:43.638335 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:43.641209 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:43.641215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:43.641230 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:43.641261 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:43.643288 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:43.643857 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:43.643891 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:43.643899 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:43.644120 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:43.644481 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:43.644484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:43.644486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:43.644916 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:43.645143 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:43.645402 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 12:38:43.645752 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260926 12:38:43.647009 14773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:43.650385 14773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:43.657145 14773 catalog_manager.cc:1347] Generated new cluster ID: d79fb1aaef64480280a46881c62d3cd8
I20260926 12:38:43.657238 14773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:43.681824 14773 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:43.683239 14773 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:43.689859 14773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c70f87a105d475ab3aa446df27c7382: Generated new TSK 0
I20260926 12:38:43.690826 14773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:43.725314 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:39135
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:43.842555 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:43.842861 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:43.842907 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:43.847711 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:43.847786 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:43.847891 14785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:43.853667 14785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:39135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:43.854866 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:43.856009 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:43.866465 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:43.866475 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:43.866472 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:43.866698 14785 server_base.cc:1034] running on GCE node
I20260926 12:38:43.867372 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:43.868124 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:43.869308 14785 hybrid_clock.cc:648] HybridClock initialized: now 1790426323869267 us; error 55 us; skew 500 ppm
Sep 26 12:38:43 dist-test-slave-m3pj krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:43.872473 14785 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:43.873479 14785 webserver.cc:460] Webserver started at http://127.10.224.129:35823/ using document root <none> and password file <none>
I20260926 12:38:43.874073 14785 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:43.874135 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:43.874399 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:43.876669 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "d14aff39b3464abca44fe98bef64b016"
format_stamp: "Formatted at 2026-09-26 12:38:43 on dist-test-slave-m3pj"
server_key: "be6e09404854631bbd370bc996dcba3a"
server_key_iv: "06446107c932405b83f6c3ee25bcb128"
server_key_version: "encryptionkey@0"
I20260926 12:38:43.877285 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "d14aff39b3464abca44fe98bef64b016"
format_stamp: "Formatted at 2026-09-26 12:38:43 on dist-test-slave-m3pj"
server_key: "be6e09404854631bbd370bc996dcba3a"
server_key_iv: "06446107c932405b83f6c3ee25bcb128"
server_key_version: "encryptionkey@0"
I20260926 12:38:43.881419 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 12:38:43.885193 14800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:43.886997 14785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:38:43.887161 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "d14aff39b3464abca44fe98bef64b016"
format_stamp: "Formatted at 2026-09-26 12:38:43 on dist-test-slave-m3pj"
server_key: "be6e09404854631bbd370bc996dcba3a"
server_key_iv: "06446107c932405b83f6c3ee25bcb128"
server_key_version: "encryptionkey@0"
I20260926 12:38:43.887327 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:43.906188 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:43.909833 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:43.910007 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:43.910656 14785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:43.911582 14785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:43.911650 14785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:43.911695 14785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:43.911716 14785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:43.921087 14785 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:37335
I20260926 12:38:43.921104 14913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:37335 every 8 connection(s)
I20260926 12:38:43.922147 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:43.922544 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 14785
I20260926 12:38:43.922672 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:43.923000 11138 external_mini_cluster.cc:1428] Setting key 9444236a627e4931971d21e3bcf69010
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
Sep 26 12:38:43 dist-test-slave-m3pj krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:43.942196 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:43.924309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:44427 (local address 127.10.224.190:39135)
0926 12:38:43.924470 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:43.924476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:43.925700 (+ 1224us) server_negotiation.cc:407] Connection header received
0926 12:38:43.927403 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:43.927407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:43.927480 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:43.927595 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:43.929498 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:43.930244 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:43.931663 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:43.932075 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:43.932203 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:43.937702 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:43.937706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:43.937708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:43.937738 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:43.939499 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:43.940267 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:43.940271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:43.940272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:43.940377 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:43.940867 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:43.940870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:43.940871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:43.941129 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:43.941247 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:43.941875 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 12:38:43.942020 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:38:43.943143 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:43.924915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:39135 (local address 127.10.224.129:44427)
0926 12:38:43.925499 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:43.925565 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:43.927163 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:43.927830 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:43.927843 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:43.928445 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:43.929272 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:43.929313 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:43.930483 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:43.930486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:43.931444 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:43.931455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:43.932246 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:43.932248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:43.932459 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:43.933187 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:43.933206 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:43.937462 (+ 4256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:43.939684 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:43.939694 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:43.939718 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:43.940135 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:43.940504 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:43.940511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:43.940516 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:43.940743 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:43.941258 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:43.941268 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:43.941605 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 12:38:43.941828 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 12:38:43.942216 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
I20260926 12:38:43.944666 14914 heartbeater.cc:347] Connected to a master server at 127.10.224.190:39135
I20260926 12:38:43.944937 14914 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:43.945540 14914 heartbeater.cc:511] Master 127.10.224.190:39135 requested a full tablet report, sending...
I20260926 12:38:43.947377 14722 ts_manager.cc:194] Registered new tserver with Master: d14aff39b3464abca44fe98bef64b016 (127.10.224.129:37335)
I20260926 12:38:43.948721 14722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:44427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:43.956311 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:39135
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:44.074461 14919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:44.074755 14919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:44.074801 14919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:44.078998 14919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:44.079078 14919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:44.079138 14919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:44.084841 14919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:39135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:44.086117 14919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:44.087361 14919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:44.099108 14924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:44.099112 14927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:44.099509 14925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:44.099440 14919 server_base.cc:1034] running on GCE node
I20260926 12:38:44.100090 14919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:44.100831 14919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:44.102085 14919 hybrid_clock.cc:648] HybridClock initialized: now 1790426324102056 us; error 47 us; skew 500 ppm
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:44.105333 14919 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:44.106496 14919 webserver.cc:460] Webserver started at http://127.10.224.130:34731/ using document root <none> and password file <none>
I20260926 12:38:44.107175 14919 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:44.107285 14919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:44.107640 14919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:44.110088 14919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "37e02ceab3fe4b8782b1ec331a278c57"
format_stamp: "Formatted at 2026-09-26 12:38:44 on dist-test-slave-m3pj"
server_key: "ad757047fdaa01dc854b40ac329158a2"
server_key_iv: "757ff57e4ce97faf0b70c8669ecd1485"
server_key_version: "encryptionkey@0"
I20260926 12:38:44.110780 14919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "37e02ceab3fe4b8782b1ec331a278c57"
format_stamp: "Formatted at 2026-09-26 12:38:44 on dist-test-slave-m3pj"
server_key: "ad757047fdaa01dc854b40ac329158a2"
server_key_iv: "757ff57e4ce97faf0b70c8669ecd1485"
server_key_version: "encryptionkey@0"
I20260926 12:38:44.115763 14919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 12:38:44.118526 14934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:44.119980 14919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:44.120139 14919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "37e02ceab3fe4b8782b1ec331a278c57"
format_stamp: "Formatted at 2026-09-26 12:38:44 on dist-test-slave-m3pj"
server_key: "ad757047fdaa01dc854b40ac329158a2"
server_key_iv: "757ff57e4ce97faf0b70c8669ecd1485"
server_key_version: "encryptionkey@0"
I20260926 12:38:44.120276 14919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:44.131309 14919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:44.135627 14919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:44.135851 14919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:44.136498 14919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:44.137391 14919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:44.137465 14919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:44.137531 14919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:44.137583 14919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:44.148293 14919 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:39929
I20260926 12:38:44.148309 15047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:39929 every 8 connection(s)
I20260926 12:38:44.149384 14919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:44.153817 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 14919
I20260926 12:38:44.153955 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:44.154244 11138 external_mini_cluster.cc:1428] Setting key 875f5a6dd7802bf6af616a8618bb7288
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:38:44.166531 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:44.151594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:49301 (local address 127.10.224.190:39135)
0926 12:38:44.151744 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:44.151748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:44.152821 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 12:38:44.154583 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:44.154587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:44.154650 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:44.154732 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:44.156267 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.156792 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.158606 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.158831 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.158942 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:44.162569 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:44.162574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:44.162576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:44.162605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:44.164197 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:44.164846 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:44.164850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:44.164851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:44.164937 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:44.165315 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:44.165318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:44.165320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:44.165525 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:44.165661 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:44.166219 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 12:38:44.166360 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:38:44.167446 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:44.152029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:39135 (local address 127.10.224.130:49301)
0926 12:38:44.152635 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:44.152685 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:44.154299 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:44.154948 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:44.154959 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:44.155506 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:44.156115 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:44.156130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.156939 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.156942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:44.158461 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:44.158477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.159012 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.159019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:44.159290 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:44.160039 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:44.160065 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:44.162303 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:44.164347 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:44.164355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:44.164375 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:44.164716 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:44.165041 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:44.165044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:44.165046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:44.165210 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:44.165661 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:44.165670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:44.165962 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 12:38:44.166163 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 12:38:44.166554 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":133,"threads_started":1}
I20260926 12:38:44.168869 15048 heartbeater.cc:347] Connected to a master server at 127.10.224.190:39135
I20260926 12:38:44.169173 15048 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:44.169742 15048 heartbeater.cc:511] Master 127.10.224.190:39135 requested a full tablet report, sending...
I20260926 12:38:44.170921 14722 ts_manager.cc:194] Registered new tserver with Master: 37e02ceab3fe4b8782b1ec331a278c57 (127.10.224.130:39929)
I20260926 12:38:44.171622 14722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:49301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:44.186728 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:39135
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:44.308338 15053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:44.308619 15053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:44.308665 15053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:44.312882 15053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:44.312949 15053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:44.313011 15053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:44.318526 15053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:39135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:44.319645 15053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:44.320667 15053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:44.330703 15059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:44.330705 15058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:44.330705 15061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:44.331180 15053 server_base.cc:1034] running on GCE node
I20260926 12:38:44.331784 15053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:44.332473 15053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:44.333636 15053 hybrid_clock.cc:648] HybridClock initialized: now 1790426324333614 us; error 31 us; skew 500 ppm
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:44.336535 15053 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:44.337584 15053 webserver.cc:460] Webserver started at http://127.10.224.131:36421/ using document root <none> and password file <none>
I20260926 12:38:44.338196 15053 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:44.338292 15053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:44.338547 15053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:44.340806 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "3a10543ca54247278f3beba9fd64c04c"
format_stamp: "Formatted at 2026-09-26 12:38:44 on dist-test-slave-m3pj"
server_key: "19e7453a3b9672b755766095cd7675e4"
server_key_iv: "0794325ce4eaf13e18e412138ff60a33"
server_key_version: "encryptionkey@0"
I20260926 12:38:44.341477 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "3a10543ca54247278f3beba9fd64c04c"
format_stamp: "Formatted at 2026-09-26 12:38:44 on dist-test-slave-m3pj"
server_key: "19e7453a3b9672b755766095cd7675e4"
server_key_iv: "0794325ce4eaf13e18e412138ff60a33"
server_key_version: "encryptionkey@0"
I20260926 12:38:44.345719 15053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:44.348452 15068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:44.349819 15053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:44.349975 15053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "3a10543ca54247278f3beba9fd64c04c"
format_stamp: "Formatted at 2026-09-26 12:38:44 on dist-test-slave-m3pj"
server_key: "19e7453a3b9672b755766095cd7675e4"
server_key_iv: "0794325ce4eaf13e18e412138ff60a33"
server_key_version: "encryptionkey@0"
I20260926 12:38:44.350111 15053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:44.364821 15053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:44.368570 15053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:44.368803 15053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:44.369448 15053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:44.370325 15053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:44.370397 15053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:44.370469 15053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:44.370522 15053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:44.380329 15053 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:35357
I20260926 12:38:44.380368 15181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:35357 every 8 connection(s)
I20260926 12:38:44.381331 15053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:44.384084 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 15053
I20260926 12:38:44.384243 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:44.384608 11138 external_mini_cluster.cc:1428] Setting key 33cd6f1011bc589d7f5c4abfe75c5fce
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:44.395948 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:44.383299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:51131 (local address 127.10.224.190:39135)
0926 12:38:44.383472 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:44.383477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:44.384526 (+ 1049us) server_negotiation.cc:407] Connection header received
0926 12:38:44.386238 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:44.386240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:44.386286 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:44.386353 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:44.387776 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.388304 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.389469 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.389689 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.389778 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:44.392373 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:44.392377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:44.392379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:44.392403 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:44.393863 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:44.394444 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:44.394447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:44.394448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:44.394552 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:44.394879 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:44.394882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:44.394883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:44.395067 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:44.395145 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:44.395703 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 12:38:44.395825 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 12:38:44.396869 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:44.383748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:39135 (local address 127.10.224.131:51131)
0926 12:38:44.384305 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:44.384372 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:44.385997 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:44.386564 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:44.386576 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:44.387048 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:44.387641 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:44.387657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.388406 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.388408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:44.389337 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:44.389347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.389841 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.389844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:44.390001 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:44.390603 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:44.390620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:44.392201 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:44.393977 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:44.393983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:44.393997 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:44.394278 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:44.394638 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:44.394640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:44.394642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:44.394781 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:44.395154 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:44.395159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:44.395403 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 12:38:44.395626 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 12:38:44.395911 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260926 12:38:44.398170 15182 heartbeater.cc:347] Connected to a master server at 127.10.224.190:39135
I20260926 12:38:44.398429 15182 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:44.399024 15182 heartbeater.cc:511] Master 127.10.224.190:39135 requested a full tablet report, sending...
I20260926 12:38:44.400107 14722 ts_manager.cc:194] Registered new tserver with Master: 3a10543ca54247278f3beba9fd64c04c (127.10.224.131:35357)
I20260926 12:38:44.400710 14722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:51131
I20260926 12:38:44.410776 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:44.437513 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:44.427370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.10.224.129:37335)
0926 12:38:44.427830 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:44.427844 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:44.427868 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:38:44.427918 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:44.427923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:44.428130 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:44.428294 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:44.429029 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.429816 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.430720 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.431119 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.431246 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:44.433848 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:44.433857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:44.433872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:44.433897 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:44.435759 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:44.436209 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:44.436214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:44.436216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:44.436296 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:44.436580 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:44.436582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:44.436583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:44.436862 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:44.437048 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:44.437221 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 12:38:44.437352 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":64,"threads_started":1}
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:44.449213 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:44.440460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.10.224.190:39135)
0926 12:38:44.440619 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:44.440624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:44.440735 (+ 111us) server_negotiation.cc:407] Connection header received
0926 12:38:44.440898 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:44.440904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:44.440971 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:44.441060 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:44.441697 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.442212 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.443016 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:44.443236 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:44.443343 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:44.445990 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:44.445993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:44.445995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:44.446019 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:44.447718 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:44.448115 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:44.448117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:44.448118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:44.448195 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:44.448427 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:44.448429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:44.448430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:44.448643 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:44.448750 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:44.448989 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 12:38:44.449090 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 12:38:44.449633 14722 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51818
I20260926 12:38:44.450261 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 14785
I20260926 12:38:44.456738 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 14919
I20260926 12:38:44.462085 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 15053
I20260926 12:38:44.467272 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 14691
2026-09-26T12:38:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2248 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[15194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[15194](info): set up 2 sockets
Sep 26 12:38:44 dist-test-slave-m3pj krb5kdc[15194](info): commencing operation
krb5kdc: starting...
W20260926 12:38:45.564175 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:45 dist-test-slave-m3pj krb5kdc[15194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:45.704082 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:46721
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:45.822154 15207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:45.822517 15207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:45.822613 15207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:45.826939 15207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:45.827042 15207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:45.827080 15207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:45.827139 15207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:45.827176 15207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:45.832873 15207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:46721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:45.834082 15207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:45.835233 15207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:45.844347 15212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:45.844341 15215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:45.844647 15207 server_base.cc:1034] running on GCE node
W20260926 12:38:45.844384 15213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:45.845495 15207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:45.846885 15207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:45.848112 15207 hybrid_clock.cc:648] HybridClock initialized: now 1790426325848096 us; error 36 us; skew 500 ppm
Sep 26 12:38:45 dist-test-slave-m3pj krb5kdc[15194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:45.851188 15207 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:45.852082 15207 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:45.852190 15207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:45.852445 15207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:45.854686 15207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "56d69a60f4ac45f3929952dc259e5008"
format_stamp: "Formatted at 2026-09-26 12:38:45 on dist-test-slave-m3pj"
server_key: "df007cc9cca24ad59524abe79829a9cf"
server_key_iv: "bed9e1541c142287ce19b546b33be075"
server_key_version: "encryptionkey@0"
I20260926 12:38:45.855317 15207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "56d69a60f4ac45f3929952dc259e5008"
format_stamp: "Formatted at 2026-09-26 12:38:45 on dist-test-slave-m3pj"
server_key: "df007cc9cca24ad59524abe79829a9cf"
server_key_iv: "bed9e1541c142287ce19b546b33be075"
server_key_version: "encryptionkey@0"
I20260926 12:38:45.859706 15207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:38:45.862434 15221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:45.863847 15207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:45.864012 15207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "56d69a60f4ac45f3929952dc259e5008"
format_stamp: "Formatted at 2026-09-26 12:38:45 on dist-test-slave-m3pj"
server_key: "df007cc9cca24ad59524abe79829a9cf"
server_key_iv: "bed9e1541c142287ce19b546b33be075"
server_key_version: "encryptionkey@0"
I20260926 12:38:45.864151 15207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:45.883842 15207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:45.887738 15207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:45.887987 15207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:45.895834 15207 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:46721
I20260926 12:38:45.896137 15273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:46721 every 8 connection(s)
I20260926 12:38:45.896634 15207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:45.900018 15274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:45.901527 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 15207
I20260926 12:38:45.901688 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:45.901993 11138 external_mini_cluster.cc:1428] Setting key f52a56e3e68860ffbf0e81cdb20383e5
I20260926 12:38:45.907142 15274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008: Bootstrap starting.
I20260926 12:38:45.909773 15274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:45.910898 15274 log.cc:824] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:38:45 dist-test-slave-m3pj krb5kdc[15194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:45.913298 15274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008: No bootstrap required, opened a new log
I20260926 12:38:45.917102 15274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d69a60f4ac45f3929952dc259e5008" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46721 } }
I20260926 12:38:45.917374 15274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:45.917428 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:45.903667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.10.224.190:46721)
0926 12:38:45.904211 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:45.904227 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:45.904266 (+ 39us) server_negotiation.cc:407] Connection header received
0926 12:38:45.904979 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:45.904998 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:45.905373 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:45.905750 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:45.906596 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:45.907700 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:45.908586 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:45.909086 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:45.909269 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:45.912510 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:45.912516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:45.912533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:45.912562 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:45.914365 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:45.914858 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:45.914863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:45.914869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:45.914995 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:45.915294 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:45.915299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:45.915301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:45.915808 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:45.916051 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:45.916277 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 12:38:45.916597 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":181,"threads_started":1}
I20260926 12:38:45.917465 15274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d69a60f4ac45f3929952dc259e5008, State: Initialized, Role: FOLLOWER
I20260926 12:38:45.918016 15274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d69a60f4ac45f3929952dc259e5008" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46721 } }
I20260926 12:38:45.918185 15274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:45.918248 15274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:45.918340 15274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:45.919353 15274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d69a60f4ac45f3929952dc259e5008" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46721 } }
I20260926 12:38:45.919796 15274 leader_election.cc:302] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d69a60f4ac45f3929952dc259e5008; no voters:
I20260926 12:38:45.920141 15274 leader_election.cc:288] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:45.920275 15279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:45.920565 15279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [term 1 LEADER]: Becoming Leader. State: Replica: 56d69a60f4ac45f3929952dc259e5008, State: Running, Role: LEADER
I20260926 12:38:45.920892 15279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d69a60f4ac45f3929952dc259e5008" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46721 } }
I20260926 12:38:45.921486 15274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:45.922607 15281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d69a60f4ac45f3929952dc259e5008. Latest consensus state: current_term: 1 leader_uuid: "56d69a60f4ac45f3929952dc259e5008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d69a60f4ac45f3929952dc259e5008" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46721 } } }
I20260926 12:38:45.922745 15281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:45.923173 15288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:45.923547 15280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d69a60f4ac45f3929952dc259e5008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d69a60f4ac45f3929952dc259e5008" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46721 } } }
I20260926 12:38:45.923655 15280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:45.926503 15288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:45.932523 15288 catalog_manager.cc:1347] Generated new cluster ID: ea69b5eb42054a9ab1cf86769f705867
I20260926 12:38:45.932621 15288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:45.940595 15288 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:45.941742 15288 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:45.958609 15288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008: Generated new TSK 0
I20260926 12:38:45.959582 15288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:45.996536 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46721
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:46.118498 15300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:46.118852 15300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:46.118937 15300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:46.123548 15300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:46.123646 15300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:46.123742 15300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:46.129412 15300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:46.130690 15300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:46.131842 15300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:46.142428 15305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:46.142442 15306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:46.142438 15308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:46.143210 15300 server_base.cc:1034] running on GCE node
I20260926 12:38:46.143796 15300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:46.144588 15300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:46.145799 15300 hybrid_clock.cc:648] HybridClock initialized: now 1790426326145756 us; error 58 us; skew 500 ppm
Sep 26 12:38:46 dist-test-slave-m3pj krb5kdc[15194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:46.148958 15300 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:46.149827 15300 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:46.149945 15300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:46.150213 15300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:46.152593 15300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "51e21f71dc0a4d44871167d809f82f17"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "3c5a27c5642c1b58fc891034e69db88b"
server_key_iv: "2df0f7e703d7fe48437f821b2f75e0ca"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.153228 15300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "51e21f71dc0a4d44871167d809f82f17"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "3c5a27c5642c1b58fc891034e69db88b"
server_key_iv: "2df0f7e703d7fe48437f821b2f75e0ca"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.157591 15300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:46.160681 15314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.162206 15300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:38:46.162374 15300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "51e21f71dc0a4d44871167d809f82f17"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "3c5a27c5642c1b58fc891034e69db88b"
server_key_iv: "2df0f7e703d7fe48437f821b2f75e0ca"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.162514 15300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:46.179086 15300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:46.182858 15300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:46.183130 15300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:46.183959 15300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:46.184973 15300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:46.185053 15300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.185135 15300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:46.185194 15300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.196110 15300 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:34131
I20260926 12:38:46.196128 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:34131 every 8 connection(s)
I20260926 12:38:46.196826 15300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:46.204741 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 15300
I20260926 12:38:46.204882 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:46.205206 11138 external_mini_cluster.cc:1428] Setting key 16700def4e063172d6a33a1eccb792a1
Sep 26 12:38:46 dist-test-slave-m3pj krb5kdc[15194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:46.212850 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.198841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:53217 (local address 127.10.224.190:46721)
0926 12:38:46.199011 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:46.199015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:46.200483 (+ 1468us) server_negotiation.cc:407] Connection header received
0926 12:38:46.201478 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:46.201482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:46.201538 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:46.201628 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:46.202951 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.203624 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.205134 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.205482 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.205600 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.208649 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:46.208653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:46.208655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:46.208682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:46.210387 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:46.211013 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:46.211017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:46.211020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:46.211112 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:46.211560 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:46.211564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:46.211565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:46.211779 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:46.211903 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:46.212497 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 12:38:46.212665 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:38:46.213896 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.199351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:46721 (local address 127.10.224.129:53217)
0926 12:38:46.200299 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:46.200346 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:46.201183 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:46.201819 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:46.201826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:46.202231 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:46.202761 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.202776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.203768 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.203771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.204947 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.204960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.205674 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.205678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.205842 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.206532 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:46.206554 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:46.208444 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:46.210538 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:46.210545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:46.210568 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:46.210890 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:46.211220 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:46.211225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:46.211227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:46.211414 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:46.211901 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:46.211909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:46.212218 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 12:38:46.212447 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 12:38:46.212865 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":198,"threads_started":1}
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:38:46.215458 15428 heartbeater.cc:347] Connected to a master server at 127.10.224.190:46721
I20260926 12:38:46.215869 15428 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:46.216434 15428 heartbeater.cc:511] Master 127.10.224.190:46721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:38:46.218374 15238 ts_manager.cc:194] Registered new tserver with Master: 51e21f71dc0a4d44871167d809f82f17 (127.10.224.129:34131)
I20260926 12:38:46.220162 15238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:53217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:46.235698 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46721
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:46.357726 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:46.358096 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:46.358151 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:46.363231 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:46.363314 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:46.363481 15433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:46.370126 15433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:46.371394 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:46.372669 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:46.383265 15441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:46.383284 15438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:46.383370 15433 server_base.cc:1034] running on GCE node
W20260926 12:38:46.383284 15439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:46.384181 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:46.384970 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:46.386171 15433 hybrid_clock.cc:648] HybridClock initialized: now 1790426326386150 us; error 35 us; skew 500 ppm
Sep 26 12:38:46 dist-test-slave-m3pj krb5kdc[15194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:46.389249 15433 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:46.390096 15433 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:46.390196 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:46.390471 15433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:46.393303 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "338cae3abd7b4b47ab027a570c043ef4"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "7eac5dedfd4300c4787fb2743f54505c"
server_key_iv: "5336fe830a066a827cb1020c20bc98ff"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.393923 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "338cae3abd7b4b47ab027a570c043ef4"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "7eac5dedfd4300c4787fb2743f54505c"
server_key_iv: "5336fe830a066a827cb1020c20bc98ff"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.398554 15433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:46.401294 15447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.402746 15433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:38:46.402894 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "338cae3abd7b4b47ab027a570c043ef4"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "7eac5dedfd4300c4787fb2743f54505c"
server_key_iv: "5336fe830a066a827cb1020c20bc98ff"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.403023 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:46.421882 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:46.425459 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:46.425675 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:46.426302 15433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:46.427260 15433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:46.427330 15433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.427402 15433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:46.427459 15433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.436725 15433 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:45063
I20260926 12:38:46.436735 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:45063 every 8 connection(s)
I20260926 12:38:46.437374 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:46.442961 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 15433
I20260926 12:38:46.443104 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:46.443404 11138 external_mini_cluster.cc:1428] Setting key 548677c7d7692aee5255985e157e7a76
Sep 26 12:38:46 dist-test-slave-m3pj krb5kdc[15194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:46.453421 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.439275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:50187 (local address 127.10.224.190:46721)
0926 12:38:46.439428 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:46.439432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:46.440416 (+ 984us) server_negotiation.cc:407] Connection header received
0926 12:38:46.441719 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:46.441722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:46.441768 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:46.441846 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:46.443106 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.443948 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.445187 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.445589 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.445722 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.448958 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:46.448966 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:46.448972 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:46.449010 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:46.450861 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:46.451452 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:46.451458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:46.451460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:46.451613 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:46.452002 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:46.452006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:46.452008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:46.452290 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:46.452440 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:46.453019 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 12:38:46.453174 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:38:46.454257 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.439772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:46721 (local address 127.10.224.130:50187)
0926 12:38:46.440251 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:46.440299 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:46.441511 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:46.442038 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:46.442045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:46.442415 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:46.442918 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.442932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.444101 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.444104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.445059 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.445070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.445786 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.445790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.446001 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.446780 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:46.446809 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:46.448740 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:46.451027 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:46.451033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:46.451050 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:46.451337 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:46.451725 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:46.451727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:46.451729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:46.451882 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:46.452465 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:46.452471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:46.452749 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 12:38:46.452930 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 12:38:46.453245 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260926 12:38:46.455605 15561 heartbeater.cc:347] Connected to a master server at 127.10.224.190:46721
I20260926 12:38:46.455883 15561 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:46.456394 15561 heartbeater.cc:511] Master 127.10.224.190:46721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:38:46.457676 15238 ts_manager.cc:194] Registered new tserver with Master: 338cae3abd7b4b47ab027a570c043ef4 (127.10.224.130:45063)
I20260926 12:38:46.458348 15238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:50187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:46.474846 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46721
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:46.590878 15566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:46.591178 15566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:46.591279 15566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:46.595603 15566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:46.595683 15566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:46.595799 15566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:46.601634 15566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:46721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:46.602963 15566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:46.604107 15566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:46.614583 15571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:46.614599 15574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:46.614599 15572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:46.614660 15566 server_base.cc:1034] running on GCE node
I20260926 12:38:46.615310 15566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:46.616148 15566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:46.617368 15566 hybrid_clock.cc:648] HybridClock initialized: now 1790426326617312 us; error 71 us; skew 500 ppm
Sep 26 12:38:46 dist-test-slave-m3pj krb5kdc[15194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:46.620445 15566 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:46.621376 15566 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:46.621488 15566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:46.621755 15566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:46.624190 15566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "9dc2beabe34b4e888bfc6486733e5008"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "8d9749e7dde126290ea7def011cd9be5"
server_key_iv: "cccb1e7139587dfbb21192cc1397749f"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.624797 15566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "9dc2beabe34b4e888bfc6486733e5008"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "8d9749e7dde126290ea7def011cd9be5"
server_key_iv: "cccb1e7139587dfbb21192cc1397749f"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.628847 15566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 12:38:46.631776 15580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.633235 15566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:38:46.633385 15566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "9dc2beabe34b4e888bfc6486733e5008"
format_stamp: "Formatted at 2026-09-26 12:38:46 on dist-test-slave-m3pj"
server_key: "8d9749e7dde126290ea7def011cd9be5"
server_key_iv: "cccb1e7139587dfbb21192cc1397749f"
server_key_version: "encryptionkey@0"
I20260926 12:38:46.633523 15566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:46.650159 15566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:46.653880 15566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:46.654103 15566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:46.654732 15566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:46.655678 15566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:46.655743 15566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.655808 15566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:46.655839 15566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:46.665580 15566 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:42675
I20260926 12:38:46.665625 15693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:42675 every 8 connection(s)
I20260926 12:38:46.666324 15566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:46.672060 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 15566
I20260926 12:38:46.672233 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:46.672533 11138 external_mini_cluster.cc:1428] Setting key a7bd63cdf7cb0c03248df4da3be7b1cf
Sep 26 12:38:46 dist-test-slave-m3pj krb5kdc[15194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426326, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:46.681481 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.668666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:43293 (local address 127.10.224.190:46721)
0926 12:38:46.668801 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:46.668806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:46.669882 (+ 1076us) server_negotiation.cc:407] Connection header received
0926 12:38:46.671249 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:46.671252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:46.671303 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:46.671399 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:46.673018 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.673550 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.674892 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.675126 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.675243 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.678022 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:46.678025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:46.678027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:46.678054 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:46.679388 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:46.679974 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:46.679976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:46.679978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:46.680058 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:46.680407 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:46.680410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:46.680411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:46.680599 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:46.680676 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:46.681218 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 12:38:46.681350 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:38:46.682425 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.669162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:46721 (local address 127.10.224.131:43293)
0926 12:38:46.669699 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:46.669745 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:46.671008 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:46.671699 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:46.671707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:46.672118 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:46.672792 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.672845 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.673694 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.673697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.674763 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.674773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.675287 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.675291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.675486 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.676174 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:46.676192 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:46.677821 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:46.679511 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:46.679516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:46.679583 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:46.679883 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:46.680146 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:46.680148 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:46.680150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:46.680297 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:46.680695 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:46.680700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:46.680966 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 12:38:46.681140 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 12:38:46.681480 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":126,"threads_started":1}
I20260926 12:38:46.683707 15694 heartbeater.cc:347] Connected to a master server at 127.10.224.190:46721
I20260926 12:38:46.683990 15694 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:46.684504 15694 heartbeater.cc:511] Master 127.10.224.190:46721 requested a full tablet report, sending...
I20260926 12:38:46.685678 15237 ts_manager.cc:194] Registered new tserver with Master: 9dc2beabe34b4e888bfc6486733e5008 (127.10.224.131:42675)
I20260926 12:38:46.686412 15237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:43293
I20260926 12:38:46.686638 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:38:46.697131 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.688805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.10.224.190:46721)
0926 12:38:46.688929 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:46.688935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:46.689173 (+ 238us) server_negotiation.cc:407] Connection header received
0926 12:38:46.689253 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:46.689256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:46.689298 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:46.689373 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:46.690200 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.690730 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.691796 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.692051 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.692149 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.693629 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:46.693633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:46.693635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:46.693661 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:46.695223 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:46.695793 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:46.695800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:46.695802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:46.695910 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:46.696234 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:46.696237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:46.696238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:46.696502 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:46.696617 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:46.696821 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 12:38:46.696922 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:38:46.701046 15237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:46.704372 15237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:46.720882 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.715480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.10.224.131:42675)
0926 12:38:46.715970 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:46.715983 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:46.716008 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:38:46.716088 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:46.716094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:46.716419 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:46.716809 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:46.716974 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.717845 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.719422 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.720094 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.720249 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.720405 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:46.720589 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 12:38:46.720692 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":138,"threads_started":1}
I20260926 12:38:46.720872 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.714778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:42675 (local address 127.0.0.1:56220)
0926 12:38:46.715181 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:46.715237 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:46.715459 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:46.716583 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:46.716589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:46.716612 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:46.716832 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.716841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.717987 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.717992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.719286 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.719304 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.720239 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.720242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.720373 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.720414 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 12:38:46.720558 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 12:38:46.720693 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":81,"spinlock_wait_cycles":3456,"thread_start_us":97,"threads_started":1}
I20260926 12:38:46.721706 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.714778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:34131 (local address 127.0.0.1:47036)
0926 12:38:46.715613 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:46.715629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:46.715721 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:46.716546 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:46.716552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:46.716580 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:46.716795 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.716803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.718573 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.718576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.720093 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.720098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.721260 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.721263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.721376 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.721398 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:38:46.721453 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 12:38:46.721539 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":248,"spinlock_wait_cycles":10496,"thread_start_us":104,"threads_started":1}
I20260926 12:38:46.721861 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.716201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:45063 (local address 127.0.0.1:59818)
0926 12:38:46.716458 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:46.716473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:46.716566 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:46.717022 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:46.717025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:46.717042 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:46.717133 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.717139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.718572 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.718576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.720084 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:46.720096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.721477 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.721481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:46.721567 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.721586 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:38:46.721660 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 12:38:46.721730 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":186,"thread_start_us":86,"threads_started":1}
I20260926 12:38:46.722784 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.714990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.10.224.129:34131)
0926 12:38:46.715972 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:46.715981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:46.716008 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:38:46.716088 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:46.716094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:46.716350 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:46.716959 (+ 609us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:46.717104 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.718420 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.720243 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.721116 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.722324 (+ 1208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.722481 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:46.722533 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:38:46.722645 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":91,"threads_started":1}
I20260926 12:38:46.722867 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:46.715834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.10.224.130:45063)
0926 12:38:46.716582 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:46.716594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:46.716615 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:38:46.716682 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:46.716687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:46.716876 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:46.717029 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:46.717254 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.718420 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.720243 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:46.721146 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:46.721261 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:46.721417 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:46.721715 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 12:38:46.722711 (+ 996us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":130,"threads_started":1}
I20260926 12:38:46.724764 15628 tablet_service.cc:1467] Processing CreateTablet for tablet f31fe6c3ade24f0d9f593b0cf90770b8 (DEFAULT_TABLE table=test-table [id=df5ecb38802b435fbbd6ca25817e5a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:46.726014 15628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31fe6c3ade24f0d9f593b0cf90770b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:46.725777 15362 tablet_service.cc:1467] Processing CreateTablet for tablet f31fe6c3ade24f0d9f593b0cf90770b8 (DEFAULT_TABLE table=test-table [id=df5ecb38802b435fbbd6ca25817e5a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:46.726835 15362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31fe6c3ade24f0d9f593b0cf90770b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:46.729301 15495 tablet_service.cc:1467] Processing CreateTablet for tablet f31fe6c3ade24f0d9f593b0cf90770b8 (DEFAULT_TABLE table=test-table [id=df5ecb38802b435fbbd6ca25817e5a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:46.730226 15495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31fe6c3ade24f0d9f593b0cf90770b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:46.733837 15711 tablet_bootstrap.cc:492] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17: Bootstrap starting.
I20260926 12:38:46.735901 15712 tablet_bootstrap.cc:492] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Bootstrap starting.
I20260926 12:38:46.736495 15711 tablet_bootstrap.cc:654] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:46.738005 15713 tablet_bootstrap.cc:492] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4: Bootstrap starting.
I20260926 12:38:46.738226 15712 tablet_bootstrap.cc:654] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:46.738397 15711 log.cc:824] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:46.739580 15712 log.cc:824] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:46.740643 15713 tablet_bootstrap.cc:654] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:46.740824 15711 tablet_bootstrap.cc:492] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17: No bootstrap required, opened a new log
I20260926 12:38:46.741057 15711 ts_tablet_manager.cc:1397] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:38:46.741480 15712 tablet_bootstrap.cc:492] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: No bootstrap required, opened a new log
I20260926 12:38:46.741513 15713 log.cc:824] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:46.741684 15712 ts_tablet_manager.cc:1397] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:38:46.743748 15713 tablet_bootstrap.cc:492] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4: No bootstrap required, opened a new log
I20260926 12:38:46.743942 15713 ts_tablet_manager.cc:1397] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:38:46.745187 15712 raft_consensus.cc:348] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:46.745476 15712 raft_consensus.cc:374] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:46.745529 15712 raft_consensus.cc:729] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc2beabe34b4e888bfc6486733e5008, State: Initialized, Role: FOLLOWER
I20260926 12:38:46.745982 15712 consensus_queue.cc:260] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:46.746193 15711 raft_consensus.cc:348] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:46.746479 15711 raft_consensus.cc:374] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:46.746548 15711 raft_consensus.cc:729] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e21f71dc0a4d44871167d809f82f17, State: Initialized, Role: FOLLOWER
I20260926 12:38:46.746869 15694 heartbeater.cc:503] Master 127.10.224.190:46721 was elected leader, sending a full tablet report...
I20260926 12:38:46.747035 15711 consensus_queue.cc:260] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:46.747156 15712 ts_tablet_manager.cc:1428] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:38:46.747785 15428 heartbeater.cc:503] Master 127.10.224.190:46721 was elected leader, sending a full tablet report...
I20260926 12:38:46.748092 15711 ts_tablet_manager.cc:1428] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 12:38:46.749171 15713 raft_consensus.cc:348] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:46.749470 15713 raft_consensus.cc:374] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:46.749596 15713 raft_consensus.cc:729] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338cae3abd7b4b47ab027a570c043ef4, State: Initialized, Role: FOLLOWER
I20260926 12:38:46.750118 15713 consensus_queue.cc:260] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:46.750959 15561 heartbeater.cc:503] Master 127.10.224.190:46721 was elected leader, sending a full tablet report...
I20260926 12:38:46.751793 15713 ts_tablet_manager.cc:1428] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260926 12:38:46.917323 15695 tablet.cc:2369] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:38:46.938570 15562 tablet.cc:2369] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:38:46.948263 15429 tablet.cc:2369] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:47.102387 15719 raft_consensus.cc:482] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:47.102609 15719 raft_consensus.cc:504] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:47.103777 15719 leader_election.cc:288] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51e21f71dc0a4d44871167d809f82f17 (127.10.224.129:34131), 9dc2beabe34b4e888bfc6486733e5008 (127.10.224.131:42675)
I20260926 12:38:47.108050 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.104280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:49505 (local address 127.10.224.129:34131)
0926 12:38:47.104422 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:47.104427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:47.104671 (+ 244us) server_negotiation.cc:407] Connection header received
0926 12:38:47.104849 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:47.104852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:47.104891 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:47.104976 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:47.105267 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.105780 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.107013 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.107423 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.107578 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:47.107656 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:47.107850 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 12:38:47.107923 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 12:38:47.108067 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.104124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:34131 (local address 127.10.224.130:49505)
0926 12:38:47.104563 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:47.104577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:47.104706 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:47.104988 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:47.104991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:47.105006 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:47.105157 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:47.105162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.105915 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.105922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:47.106891 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:47.106900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.107573 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.107577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:47.107684 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:47.107711 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:38:47.107815 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 12:38:47.107896 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260926 12:38:47.108222 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.104265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:33723 (local address 127.10.224.131:42675)
0926 12:38:47.104413 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:47.104420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:47.104734 (+ 314us) server_negotiation.cc:407] Connection header received
0926 12:38:47.104899 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:47.104902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:47.104951 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:47.105032 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:47.105362 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.105853 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.107107 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.107596 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.107768 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:47.107839 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:47.108006 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 12:38:47.108073 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:38:47.108721 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.104123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:42675 (local address 127.10.224.130:33723)
0926 12:38:47.104599 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:47.104611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:47.104778 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:47.105053 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:47.105055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:47.105071 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:47.105244 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:47.105250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.105984 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.105988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:47.106960 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:47.106995 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.107756 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.107759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:47.107865 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:47.107894 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:38:47.107981 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:38:47.108054 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":226,"thread_start_us":89,"threads_started":1}
I20260926 12:38:47.108745 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31fe6c3ade24f0d9f593b0cf90770b8" candidate_uuid: "338cae3abd7b4b47ab027a570c043ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e21f71dc0a4d44871167d809f82f17" is_pre_election: true
I20260926 12:38:47.109107 15382 raft_consensus.cc:2454] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 338cae3abd7b4b47ab027a570c043ef4 in term 0.
I20260926 12:38:47.109397 15648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31fe6c3ade24f0d9f593b0cf90770b8" candidate_uuid: "338cae3abd7b4b47ab027a570c043ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc2beabe34b4e888bfc6486733e5008" is_pre_election: true
I20260926 12:38:47.109699 15648 raft_consensus.cc:2454] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 338cae3abd7b4b47ab027a570c043ef4 in term 0.
I20260926 12:38:47.109674 15451 leader_election.cc:302] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338cae3abd7b4b47ab027a570c043ef4, 51e21f71dc0a4d44871167d809f82f17; no voters:
I20260926 12:38:47.109980 15719 raft_consensus.cc:2781] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:47.110088 15719 raft_consensus.cc:482] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:47.110155 15719 raft_consensus.cc:3037] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:47.111234 15719 raft_consensus.cc:504] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:47.111641 15719 leader_election.cc:288] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [CANDIDATE]: Term 1 election: Requested vote from peers 51e21f71dc0a4d44871167d809f82f17 (127.10.224.129:34131), 9dc2beabe34b4e888bfc6486733e5008 (127.10.224.131:42675)
I20260926 12:38:47.112030 15648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31fe6c3ade24f0d9f593b0cf90770b8" candidate_uuid: "338cae3abd7b4b47ab027a570c043ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc2beabe34b4e888bfc6486733e5008"
I20260926 12:38:47.112169 15648 raft_consensus.cc:3037] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:47.112169 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31fe6c3ade24f0d9f593b0cf90770b8" candidate_uuid: "338cae3abd7b4b47ab027a570c043ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e21f71dc0a4d44871167d809f82f17"
I20260926 12:38:47.112323 15382 raft_consensus.cc:3037] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:47.113318 15648 raft_consensus.cc:2454] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 338cae3abd7b4b47ab027a570c043ef4 in term 1.
I20260926 12:38:47.113416 15382 raft_consensus.cc:2454] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 338cae3abd7b4b47ab027a570c043ef4 in term 1.
I20260926 12:38:47.113667 15450 leader_election.cc:302] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338cae3abd7b4b47ab027a570c043ef4, 9dc2beabe34b4e888bfc6486733e5008; no voters:
I20260926 12:38:47.113855 15719 raft_consensus.cc:2781] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:47.114065 15719 raft_consensus.cc:686] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 1 LEADER]: Becoming Leader. State: Replica: 338cae3abd7b4b47ab027a570c043ef4, State: Running, Role: LEADER
I20260926 12:38:47.114431 15719 consensus_queue.cc:237] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } }
I20260926 12:38:47.117856 15236 catalog_manager.cc:5495] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 338cae3abd7b4b47ab027a570c043ef4 (127.10.224.130). New cstate: current_term: 1 leader_uuid: "338cae3abd7b4b47ab027a570c043ef4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "338cae3abd7b4b47ab027a570c043ef4" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 45063 } health_report { overall_health: HEALTHY } } }
I20260926 12:38:47.176024 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.172315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.10.224.130:45063)
0926 12:38:47.172467 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:47.172471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:47.172484 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:38:47.172567 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:47.172570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:47.172638 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:47.172729 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:47.172973 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.173531 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.174464 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:47.174688 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:47.174770 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:47.175058 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:47.175591 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:47.175684 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:47.175773 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 12:38:47.175831 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:38:47.182729 15382 raft_consensus.cc:1261] T f31fe6c3ade24f0d9f593b0cf90770b8 P 51e21f71dc0a4d44871167d809f82f17 [term 1 FOLLOWER]: Refusing update from remote peer 338cae3abd7b4b47ab027a570c043ef4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:47.182775 15648 raft_consensus.cc:1261] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 1 FOLLOWER]: Refusing update from remote peer 338cae3abd7b4b47ab027a570c043ef4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:47.183638 15722 consensus_queue.cc:1035] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc2beabe34b4e888bfc6486733e5008" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 42675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:47.183876 15719 consensus_queue.cc:1035] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51e21f71dc0a4d44871167d809f82f17" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 34131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:47.195209 15727 mvcc.cc:205] Tried to move back new op lower bound from 7333586236132917248 to 7333586235863547904. Current Snapshot: MvccSnapshot[applied={T|T < 7333586236132917248}]
I20260926 12:38:47.203089 15236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:47.203299 15236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:47.206352 15236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d69a60f4ac45f3929952dc259e5008: Sending DeleteTablet for 3 replicas of tablet f31fe6c3ade24f0d9f593b0cf90770b8
I20260926 12:38:47.207334 15362 tablet_service.cc:1514] Processing DeleteTablet for tablet f31fe6c3ade24f0d9f593b0cf90770b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:47 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:47036
I20260926 12:38:47.207391 15628 tablet_service.cc:1514] Processing DeleteTablet for tablet f31fe6c3ade24f0d9f593b0cf90770b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:47 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:56220
I20260926 12:38:47.207386 15495 tablet_service.cc:1514] Processing DeleteTablet for tablet f31fe6c3ade24f0d9f593b0cf90770b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:47 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:59818
I20260926 12:38:47.208236 15736 tablet_replica.cc:330] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4: stopping tablet replica
I20260926 12:38:47.208758 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 15300
I20260926 12:38:47.209583 15734 tablet_replica.cc:330] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: stopping tablet replica
I20260926 12:38:47.209908 15734 raft_consensus.cc:2229] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:47.210248 15734 raft_consensus.cc:2258] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:47.210968 15736 raft_consensus.cc:2229] T f31fe6c3ade24f0d9f593b0cf90770b8 P 338cae3abd7b4b47ab027a570c043ef4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:47.212790 15734 ts_tablet_manager.cc:1905] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 12:38:47.215150 15225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:34131 (error 108)
I20260926 12:38:47.215734 15734 ts_tablet_manager.cc:1918] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:47.215837 15734 log.cc:1196] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/f31fe6c3ade24f0d9f593b0cf90770b8
I20260926 12:38:47.215844 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 15433
I20260926 12:38:47.216126 15734 ts_tablet_manager.cc:1939] T f31fe6c3ade24f0d9f593b0cf90770b8 P 9dc2beabe34b4e888bfc6486733e5008: Deleting consensus metadata
I20260926 12:38:47.217587 15224 catalog_manager.cc:4841] TS 9dc2beabe34b4e888bfc6486733e5008 (127.10.224.131:42675): tablet f31fe6c3ade24f0d9f593b0cf90770b8 (table test-table [id=df5ecb38802b435fbbd6ca25817e5a93]) successfully deleted
I20260926 12:38:47.218391 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.218017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:34131 (local address 127.0.0.1:47052)
0926 12:38:47.218192 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:47.218285 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:34131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 12:38:47.218580 15225 catalog_manager.cc:4584] TS 51e21f71dc0a4d44871167d809f82f17 (127.10.224.129:34131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f31fe6c3ade24f0d9f593b0cf90770b8: Network error: Client connection negotiation failed: client connection to 127.10.224.129:34131: connect: Connection refused (error 111)
I20260926 12:38:47.222321 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 15566
I20260926 12:38:47.224180 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.222505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:45063 (local address 127.0.0.1:59834)
0926 12:38:47.224045 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:47.224102 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.130:45063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1451}
I20260926 12:38:47.228826 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 15207
2026-09-26T12:38:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2762 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T12:38:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:47Z Disabled control of system clock
I20260926 12:38:47.262856 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35813
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:35813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 12:38:47.382329 15741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:47.382716 15741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:47.382799 15741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:47.387135 15741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:47.387274 15741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:47.387315 15741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:47.387354 15741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:47.387387 15741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:47.392659 15741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:35813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:47.393951 15741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:47.395474 15741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:47.405738 15749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:47.405823 15741 server_base.cc:1034] running on GCE node
W20260926 12:38:47.405743 15746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:47.405791 15747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:47.406924 15741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:47.408445 15741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:47.409660 15741 hybrid_clock.cc:648] HybridClock initialized: now 1790426327409618 us; error 56 us; skew 500 ppm
I20260926 12:38:47.412168 15741 webserver.cc:460] Webserver started at http://127.10.224.190:39033/ using document root <none> and password file <none>
I20260926 12:38:47.412848 15741 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:47.412920 15741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:47.413164 15741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:47.415458 15741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "0e88c80c39704a5d96971505d9815cc9"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "44c91160519a74e4c1d77fa014debb3d"
server_key_iv: "f24fd9dda274c88ecfb8721b84df0a9c"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.416210 15741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "0e88c80c39704a5d96971505d9815cc9"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "44c91160519a74e4c1d77fa014debb3d"
server_key_iv: "f24fd9dda274c88ecfb8721b84df0a9c"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.420549 15741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:47.423346 15755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:47.424801 15741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:47.424968 15741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "0e88c80c39704a5d96971505d9815cc9"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "44c91160519a74e4c1d77fa014debb3d"
server_key_iv: "f24fd9dda274c88ecfb8721b84df0a9c"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.425105 15741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:47.430172 15741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:47.430864 15741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:47.431077 15741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:47.438711 15741 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:35813
I20260926 12:38:47.438791 15807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:35813 every 8 connection(s)
I20260926 12:38:47.439926 15741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:47.443472 15808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:47.449182 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 15741
I20260926 12:38:47.449316 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:47.449615 11138 external_mini_cluster.cc:1428] Setting key 6ee33b4a7bb05eceebfd558a3ef49117
I20260926 12:38:47.449738 15808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9: Bootstrap starting.
I20260926 12:38:47.452445 15808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:47.453253 15808 log.cc:824] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:47.455473 15808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9: No bootstrap required, opened a new log
I20260926 12:38:47.457330 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.451116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.10.224.190:35813)
0926 12:38:47.451907 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:47.451925 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:47.451971 (+ 46us) server_negotiation.cc:407] Connection header received
0926 12:38:47.452732 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:47.452753 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:47.453157 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:47.453576 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:47.455915 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:47.455920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:47.455933 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:47.455945 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:47.456020 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:47.456037 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:47.456142 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:47.456282 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 12:38:47.456610 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":170,"threads_started":1}
I20260926 12:38:47.459455 15808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e88c80c39704a5d96971505d9815cc9" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35813 } }
I20260926 12:38:47.459777 15808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:47.459854 15808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e88c80c39704a5d96971505d9815cc9, State: Initialized, Role: FOLLOWER
I20260926 12:38:47.460281 15808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e88c80c39704a5d96971505d9815cc9" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35813 } }
I20260926 12:38:47.460428 15808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:47.460521 15808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:47.460646 15808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:47.461939 15808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e88c80c39704a5d96971505d9815cc9" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35813 } }
I20260926 12:38:47.462357 15808 leader_election.cc:302] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e88c80c39704a5d96971505d9815cc9; no voters:
I20260926 12:38:47.462819 15808 leader_election.cc:288] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:47.462929 15813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:47.463171 15813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e88c80c39704a5d96971505d9815cc9, State: Running, Role: LEADER
I20260926 12:38:47.463502 15813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e88c80c39704a5d96971505d9815cc9" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35813 } }
I20260926 12:38:47.464222 15808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:47.465375 15814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e88c80c39704a5d96971505d9815cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e88c80c39704a5d96971505d9815cc9" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35813 } } }
I20260926 12:38:47.465504 15814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:47.465734 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e88c80c39704a5d96971505d9815cc9. Latest consensus state: current_term: 1 leader_uuid: "0e88c80c39704a5d96971505d9815cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e88c80c39704a5d96971505d9815cc9" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35813 } } }
I20260926 12:38:47.465822 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:47.465969 15822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:47.468899 15822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:47.474910 15822 catalog_manager.cc:1347] Generated new cluster ID: df14eab186184ce79c9c14be1e86a425
I20260926 12:38:47.474993 15822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:47.491531 15822 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:47.491906 15822 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:47.502028 15822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9: Generated new TSK 0
I20260926 12:38:47.502982 15822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:38:47.515071 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35813
--builtin_ntp_servers=127.10.224.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 12:38:47.638599 15832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:47.638918 15832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:47.638960 15832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:47.643383 15832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:47.643461 15832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:47.643586 15832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:47.648710 15832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:47.649955 15832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:47.651507 15832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:47.662232 15840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:47.662261 15837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:47.662251 15838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:47.662305 15832 server_base.cc:1034] running on GCE node
I20260926 12:38:47.662907 15832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:47.663736 15832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:47.665030 15832 hybrid_clock.cc:648] HybridClock initialized: now 1790426327664999 us; error 35 us; skew 500 ppm
I20260926 12:38:47.667163 15832 webserver.cc:460] Webserver started at http://127.10.224.129:45973/ using document root <none> and password file <none>
I20260926 12:38:47.667860 15832 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:47.667934 15832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:47.668123 15832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:47.670333 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "46a85e16f7cc4b28ad6beb683b81b139"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "33b48533c1c3096e40fe916829b448bc"
server_key_iv: "6c867f59f71a46aff4e2acc58cca8325"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.670902 15832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "46a85e16f7cc4b28ad6beb683b81b139"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "33b48533c1c3096e40fe916829b448bc"
server_key_iv: "6c867f59f71a46aff4e2acc58cca8325"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.675148 15832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:47.678224 15846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:47.679838 15832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:47.679958 15832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "46a85e16f7cc4b28ad6beb683b81b139"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "33b48533c1c3096e40fe916829b448bc"
server_key_iv: "6c867f59f71a46aff4e2acc58cca8325"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.680090 15832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:47.685007 15832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:47.685662 15832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:47.685809 15832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:47.686365 15832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:47.687281 15832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:47.687328 15832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:47.687368 15832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:47.687391 15832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:47.696235 15832 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:42231
I20260926 12:38:47.696257 15959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:42231 every 8 connection(s)
I20260926 12:38:47.697224 15832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:47.701866 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 15832
I20260926 12:38:47.701972 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:47.702280 11138 external_mini_cluster.cc:1428] Setting key 199eaf19ebe923446ad4bb42039e6296
I20260926 12:38:47.704382 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.699387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:52135 (local address 127.10.224.190:35813)
0926 12:38:47.699605 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:47.699614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:47.700726 (+ 1112us) server_negotiation.cc:407] Connection header received
0926 12:38:47.702230 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:47.702234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:47.702280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:47.702396 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:47.703717 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:47.703720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:47.703722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:47.703855 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:47.703882 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:47.703893 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:47.703988 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:47.704181 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 12:38:47.704225 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:38:47.704577 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35813
--builtin_ntp_servers=127.10.224.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 12:38:47.705430 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.699874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35813 (local address 127.10.224.129:52135)
0926 12:38:47.700564 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:47.700606 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:47.701993 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:47.702578 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:47.702585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:47.702715 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:47.703390 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:38:47.703392 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:47.703435 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:38:47.703507 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:47.704006 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:47.704015 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:47.704020 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:38:47.704192 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 12:38:47.704423 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260926 12:38:47.706805 15960 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35813
I20260926 12:38:47.707086 15960 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:47.707623 15960 heartbeater.cc:511] Master 127.10.224.190:35813 requested a full tablet report, sending...
I20260926 12:38:47.709312 15771 ts_manager.cc:194] Registered new tserver with Master: 46a85e16f7cc4b28ad6beb683b81b139 (127.10.224.129:42231)
W20260926 12:38:47.821430 15963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:47.821803 15963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:47.821889 15963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:47.826217 15963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:47.826325 15963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:47.826431 15963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:47.831840 15963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:47.833055 15963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:47.834548 15963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:47.845041 15968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:47.845018 15971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:47.844997 15969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:47.845250 15963 server_base.cc:1034] running on GCE node
I20260926 12:38:47.845916 15963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:47.846676 15963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:47.847849 15963 hybrid_clock.cc:648] HybridClock initialized: now 1790426327847806 us; error 60 us; skew 500 ppm
I20260926 12:38:47.849869 15963 webserver.cc:460] Webserver started at http://127.10.224.130:44715/ using document root <none> and password file <none>
I20260926 12:38:47.850512 15963 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:47.850582 15963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:47.850760 15963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:47.853138 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "d7ee7e67e149461683cdbf8ac50d302e"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "c56a6b5029f91c5168eaee08113481ef"
server_key_iv: "38e5741e60a267ae2c9fd31c3179b921"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.853713 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "d7ee7e67e149461683cdbf8ac50d302e"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "c56a6b5029f91c5168eaee08113481ef"
server_key_iv: "38e5741e60a267ae2c9fd31c3179b921"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.857940 15963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:47.861326 15977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:47.862943 15963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:38:47.863085 15963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "d7ee7e67e149461683cdbf8ac50d302e"
format_stamp: "Formatted at 2026-09-26 12:38:47 on dist-test-slave-m3pj"
server_key: "c56a6b5029f91c5168eaee08113481ef"
server_key_iv: "38e5741e60a267ae2c9fd31c3179b921"
server_key_version: "encryptionkey@0"
I20260926 12:38:47.863226 15963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:47.868822 15963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:47.869539 15963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:47.869745 15963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:47.870358 15963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:47.871385 15963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:47.871435 15963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:47.871485 15963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:47.871622 15963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 12:38:47.881037 15963 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:35047
I20260926 12:38:47.881054 16090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:35047 every 8 connection(s)
I20260926 12:38:47.882117 15963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:47.889528 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.884486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:54129 (local address 127.10.224.190:35813)
0926 12:38:47.884639 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:47.884652 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:47.885908 (+ 1256us) server_negotiation.cc:407] Connection header received
0926 12:38:47.887377 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:47.887381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:47.887434 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:47.887606 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:47.888955 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:47.888959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:47.888962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:47.888983 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:47.889004 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:47.889013 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:47.889114 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:47.889262 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 12:38:47.889312 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:38:47.890470 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:47.884962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35813 (local address 127.10.224.130:54129)
0926 12:38:47.885701 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:47.885747 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:47.887111 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:47.887818 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:47.887826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:47.887941 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:47.888643 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:38:47.888645 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:47.888690 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:38:47.888768 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:47.889102 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:47.889111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:47.889118 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:38:47.889315 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 12:38:47.889598 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
I20260926 12:38:47.891942 16091 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35813
I20260926 12:38:47.892196 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 15963
I20260926 12:38:47.892230 16091 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:47.892314 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:47.892635 11138 external_mini_cluster.cc:1428] Setting key ef40417a03d3367b42c0c4223b1eabc5
I20260926 12:38:47.892737 16091 heartbeater.cc:511] Master 127.10.224.190:35813 requested a full tablet report, sending...
I20260926 12:38:47.894011 15772 ts_manager.cc:194] Registered new tserver with Master: d7ee7e67e149461683cdbf8ac50d302e (127.10.224.130:35047)
I20260926 12:38:47.894882 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35813
--builtin_ntp_servers=127.10.224.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 12:38:48.014425 16094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:48.014720 16094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:48.014762 16094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:48.019055 16094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:48.019132 16094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:48.019220 16094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:48.024439 16094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:48.025631 16094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:48.027127 16094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:48.037537 16099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:48.037537 16102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:48.037694 16100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:48.037817 16094 server_base.cc:1034] running on GCE node
I20260926 12:38:48.038427 16094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:48.039083 16094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:48.040306 16094 hybrid_clock.cc:648] HybridClock initialized: now 1790426328040281 us; error 36 us; skew 500 ppm
I20260926 12:38:48.042613 16094 webserver.cc:460] Webserver started at http://127.10.224.131:38997/ using document root <none> and password file <none>
I20260926 12:38:48.043286 16094 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:48.043360 16094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:48.043648 16094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:48.046160 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "a4892b3ea7224daeb606964f6a058980"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "18544cdf1dda2f8c12151113a2bc18dc"
server_key_iv: "5f06eaa5188435875b809c3b1fbd1a86"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.046824 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "a4892b3ea7224daeb606964f6a058980"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "18544cdf1dda2f8c12151113a2bc18dc"
server_key_iv: "5f06eaa5188435875b809c3b1fbd1a86"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.051373 16094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 12:38:48.054419 16108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:48.056016 16094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:48.056183 16094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "a4892b3ea7224daeb606964f6a058980"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "18544cdf1dda2f8c12151113a2bc18dc"
server_key_iv: "5f06eaa5188435875b809c3b1fbd1a86"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.056316 16094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:48.061611 16094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:48.062278 16094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:48.062489 16094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:48.063153 16094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:48.064220 16094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:48.064265 16094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:48.064347 16094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:48.064379 16094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:48.074258 16094 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:38367
I20260926 12:38:48.074359 16221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:38367 every 8 connection(s)
I20260926 12:38:48.075292 16094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:48.082165 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 16094
I20260926 12:38:48.082305 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:48.082296 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.077471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:39017 (local address 127.10.224.190:35813)
0926 12:38:48.077664 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.077669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.078796 (+ 1127us) server_negotiation.cc:407] Connection header received
0926 12:38:48.080147 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.080150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.080190 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.080272 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.081581 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.081585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.081587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.081600 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.081621 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.081629 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.081724 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.082089 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.082136 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 12:38:48.082595 11138 external_mini_cluster.cc:1428] Setting key 327e66f537f005a6383f3b39889632f6
I20260926 12:38:48.083428 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.077891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35813 (local address 127.10.224.131:39017)
0926 12:38:48.078613 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:48.078672 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:48.079931 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:48.080451 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:48.080458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:48.080558 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:48.081122 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:38:48.081124 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:48.081169 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:38:48.081246 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:48.081881 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:48.081893 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:48.081902 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:38:48.082102 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 12:38:48.082409 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260926 12:38:48.084709 16222 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35813
I20260926 12:38:48.085000 16222 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:48.085465 16222 heartbeater.cc:511] Master 127.10.224.190:35813 requested a full tablet report, sending...
I20260926 12:38:48.086553 15771 ts_manager.cc:194] Registered new tserver with Master: a4892b3ea7224daeb606964f6a058980 (127.10.224.131:38367)
I20260926 12:38:48.097272 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:38:48.101586 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.099613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.10.224.190:35813)
0926 12:38:48.099781 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.099786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.099950 (+ 164us) server_negotiation.cc:407] Connection header received
0926 12:38:48.100114 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.100118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.100166 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.100266 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.101100 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.101104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.101108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.101129 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.101153 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.101162 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.101274 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.101374 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.101430 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:38:48.105451 15771 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:48.108667 15771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:48.122385 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.118956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:35047 (local address 127.0.0.1:48022)
0926 12:38:48.119720 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:48.119735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:48.119815 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:48.120574 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:48.120580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:48.120668 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:48.121335 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:38:48.121338 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:48.121362 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:38:48.121442 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:48.122002 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:48.122012 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:48.122019 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:38:48.122153 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 12:38:48.122237 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":94,"threads_started":1}
I20260926 12:38:48.122586 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.119440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38367 (local address 127.0.0.1:51320)
0926 12:38:48.119819 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:48.119830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:48.119910 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:48.121015 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:48.121019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:48.121036 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:48.121909 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:38:48.121911 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:48.121924 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:38:48.121957 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:48.122346 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:48.122349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:48.122352 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 12:38:48.122418 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:38:48.122456 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":108,"threads_started":1}
I20260926 12:38:48.122643 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.119662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.10.224.131:38367)
0926 12:38:48.120077 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.120090 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.120113 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:38:48.120176 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.120182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.120383 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.120529 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.122099 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.122106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.122118 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.122132 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.122178 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.122233 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.122376 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.122428 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.122503 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":86,"threads_started":1}
I20260926 12:38:48.123137 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.119560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.10.224.130:35047)
0926 12:38:48.120155 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.120166 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.120185 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:38:48.120229 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.120233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.120377 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.120529 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.121692 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.121699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.121714 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.121726 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.121778 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.121817 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.122018 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.122175 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.122975 (+ 800us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":112,"threads_started":1}
I20260926 12:38:48.123692 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.118559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42231 (local address 127.0.0.1:48966)
0926 12:38:48.119353 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:48.119412 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:48.119883 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:48.121242 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:48.121248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:48.121560 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:48.122964 (+ 1404us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:38:48.122966 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:48.122976 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:38:48.123002 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:48.123427 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:48.123430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:48.123433 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 12:38:48.123543 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 12:38:48.123579 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":120,"threads_started":1}
I20260926 12:38:48.123929 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.118755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.10.224.129:42231)
0926 12:38:48.120058 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.120068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.120090 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:38:48.120140 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.120145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.120320 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.120593 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.123127 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.123136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.123152 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.123168 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.123216 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.123262 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.123379 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.123577 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.123651 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":155,"threads_started":1}
I20260926 12:38:48.125902 16156 tablet_service.cc:1467] Processing CreateTablet for tablet 1fba7b6a97b749ff94875f25c59d45a6 (DEFAULT_TABLE table=test-table [id=eb5ce8a244724385b5bd16cdf7b90f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:48.125909 16025 tablet_service.cc:1467] Processing CreateTablet for tablet 1fba7b6a97b749ff94875f25c59d45a6 (DEFAULT_TABLE table=test-table [id=eb5ce8a244724385b5bd16cdf7b90f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:48.126995 16025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fba7b6a97b749ff94875f25c59d45a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:48.127030 16156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fba7b6a97b749ff94875f25c59d45a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:48.126721 15894 tablet_service.cc:1467] Processing CreateTablet for tablet 1fba7b6a97b749ff94875f25c59d45a6 (DEFAULT_TABLE table=test-table [id=eb5ce8a244724385b5bd16cdf7b90f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:48.127712 15894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fba7b6a97b749ff94875f25c59d45a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:48.134076 16239 tablet_bootstrap.cc:492] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: Bootstrap starting.
I20260926 12:38:48.134997 16241 tablet_bootstrap.cc:492] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Bootstrap starting.
I20260926 12:38:48.135571 16240 tablet_bootstrap.cc:492] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139: Bootstrap starting.
I20260926 12:38:48.137316 16239 tablet_bootstrap.cc:654] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:48.137405 16241 tablet_bootstrap.cc:654] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:48.137450 16240 tablet_bootstrap.cc:654] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:48.138128 16239 log.cc:824] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:48.138289 16241 log.cc:824] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:48.138289 16240 log.cc:824] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:48.140321 16241 tablet_bootstrap.cc:492] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: No bootstrap required, opened a new log
I20260926 12:38:48.140331 16239 tablet_bootstrap.cc:492] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: No bootstrap required, opened a new log
I20260926 12:38:48.140547 16239 ts_tablet_manager.cc:1397] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 12:38:48.140547 16241 ts_tablet_manager.cc:1397] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:38:48.144404 16239 raft_consensus.cc:348] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.144403 16241 raft_consensus.cc:348] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.144687 16239 raft_consensus.cc:374] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:48.144680 16241 raft_consensus.cc:374] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:48.144755 16241 raft_consensus.cc:729] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4892b3ea7224daeb606964f6a058980, State: Initialized, Role: FOLLOWER
I20260926 12:38:48.144757 16239 raft_consensus.cc:729] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ee7e67e149461683cdbf8ac50d302e, State: Initialized, Role: FOLLOWER
I20260926 12:38:48.145237 16241 consensus_queue.cc:260] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.145243 16239 consensus_queue.cc:260] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.146227 16222 heartbeater.cc:503] Master 127.10.224.190:35813 was elected leader, sending a full tablet report...
I20260926 12:38:48.146420 16239 ts_tablet_manager.cc:1428] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:38:48.146430 16241 ts_tablet_manager.cc:1428] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:38:48.146536 16240 tablet_bootstrap.cc:492] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139: No bootstrap required, opened a new log
I20260926 12:38:48.146761 16091 heartbeater.cc:503] Master 127.10.224.190:35813 was elected leader, sending a full tablet report...
I20260926 12:38:48.146750 16240 ts_tablet_manager.cc:1397] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260926 12:38:48.151358 16240 raft_consensus.cc:348] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.151749 16240 raft_consensus.cc:374] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:48.151815 16240 raft_consensus.cc:729] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a85e16f7cc4b28ad6beb683b81b139, State: Initialized, Role: FOLLOWER
I20260926 12:38:48.152311 16240 consensus_queue.cc:260] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.153076 15960 heartbeater.cc:503] Master 127.10.224.190:35813 was elected leader, sending a full tablet report...
I20260926 12:38:48.153352 16240 ts_tablet_manager.cc:1428] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260926 12:38:48.199414 15961 tablet.cc:2369] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:48.319238 16246 raft_consensus.cc:482] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:48.319511 16246 raft_consensus.cc:504] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.320822 16246 leader_election.cc:288] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46a85e16f7cc4b28ad6beb683b81b139 (127.10.224.129:42231), a4892b3ea7224daeb606964f6a058980 (127.10.224.131:38367)
I20260926 12:38:48.323395 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.321210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42231 (local address 127.10.224.130:44121)
0926 12:38:48.321388 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:48.321406 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:48.321546 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:48.321984 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:48.321987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:48.322001 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:48.322598 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:38:48.322602 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:48.322615 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:38:48.322652 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:48.323047 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:48.323053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:48.323063 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 12:38:48.323192 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 12:38:48.323237 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 12:38:48.323549 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.321321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:44121 (local address 127.10.224.129:42231)
0926 12:38:48.321528 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.321536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.321558 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:38:48.321714 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.321721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.321775 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.321875 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.322859 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.322863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.322867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.322882 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.322907 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.322918 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.323025 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.323224 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.323302 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:38:48.323915 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.321775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:58615 (local address 127.10.224.131:38367)
0926 12:38:48.321902 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.321906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.322145 (+ 239us) server_negotiation.cc:407] Connection header received
0926 12:38:48.322240 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.322245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.322289 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.322387 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.323343 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.323347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.323350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.323364 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.323388 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.323399 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.323503 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.323714 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.323765 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:38:48.324231 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.321630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38367 (local address 127.10.224.130:58615)
0926 12:38:48.321996 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:48.322030 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:48.322130 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:48.322513 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:48.322520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:48.322555 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:48.323142 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:38:48.323145 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:48.323161 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:38:48.323192 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:48.323574 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:48.323579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:48.323582 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 12:38:48.324077 (+ 495us) client_negotiation.cc:240] Negotiation successful
0926 12:38:48.324123 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":150,"threads_started":1}
I20260926 12:38:48.324258 15914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fba7b6a97b749ff94875f25c59d45a6" candidate_uuid: "d7ee7e67e149461683cdbf8ac50d302e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a85e16f7cc4b28ad6beb683b81b139" is_pre_election: true
I20260926 12:38:48.324651 15914 raft_consensus.cc:2454] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ee7e67e149461683cdbf8ac50d302e in term 0.
I20260926 12:38:48.325148 15980 leader_election.cc:302] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a85e16f7cc4b28ad6beb683b81b139, d7ee7e67e149461683cdbf8ac50d302e; no voters:
I20260926 12:38:48.325467 16246 raft_consensus.cc:2781] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:48.325459 16176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fba7b6a97b749ff94875f25c59d45a6" candidate_uuid: "d7ee7e67e149461683cdbf8ac50d302e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4892b3ea7224daeb606964f6a058980" is_pre_election: true
I20260926 12:38:48.325572 16246 raft_consensus.cc:482] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:48.325619 16246 raft_consensus.cc:3037] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:48.325773 16176 raft_consensus.cc:2454] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ee7e67e149461683cdbf8ac50d302e in term 0.
W20260926 12:38:48.326375 16223 tablet.cc:2369] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:48.327081 16246 raft_consensus.cc:504] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.327406 16246 leader_election.cc:288] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [CANDIDATE]: Term 1 election: Requested vote from peers 46a85e16f7cc4b28ad6beb683b81b139 (127.10.224.129:42231), a4892b3ea7224daeb606964f6a058980 (127.10.224.131:38367)
I20260926 12:38:48.327973 16176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fba7b6a97b749ff94875f25c59d45a6" candidate_uuid: "d7ee7e67e149461683cdbf8ac50d302e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4892b3ea7224daeb606964f6a058980"
I20260926 12:38:48.327963 15914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fba7b6a97b749ff94875f25c59d45a6" candidate_uuid: "d7ee7e67e149461683cdbf8ac50d302e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a85e16f7cc4b28ad6beb683b81b139"
I20260926 12:38:48.328084 16176 raft_consensus.cc:3037] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:48.328122 15914 raft_consensus.cc:3037] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:48.329340 15914 raft_consensus.cc:2454] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ee7e67e149461683cdbf8ac50d302e in term 1.
I20260926 12:38:48.329376 16176 raft_consensus.cc:2454] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ee7e67e149461683cdbf8ac50d302e in term 1.
I20260926 12:38:48.329696 15980 leader_election.cc:302] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a85e16f7cc4b28ad6beb683b81b139, d7ee7e67e149461683cdbf8ac50d302e; no voters:
I20260926 12:38:48.329999 16246 raft_consensus.cc:2781] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:48.330255 16246 raft_consensus.cc:686] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 1 LEADER]: Becoming Leader. State: Replica: d7ee7e67e149461683cdbf8ac50d302e, State: Running, Role: LEADER
I20260926 12:38:48.330650 16246 consensus_queue.cc:237] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } }
I20260926 12:38:48.334363 15772 catalog_manager.cc:5495] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ee7e67e149461683cdbf8ac50d302e (127.10.224.130). New cstate: current_term: 1 leader_uuid: "d7ee7e67e149461683cdbf8ac50d302e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ee7e67e149461683cdbf8ac50d302e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35047 } health_report { overall_health: HEALTHY } } }
I20260926 12:38:48.364967 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.362942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.10.224.130:35047)
0926 12:38:48.363107 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.363111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.363127 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:38:48.363214 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.363217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.363284 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.363376 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.364416 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.364419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.364421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.364436 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.364457 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.364466 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.364551 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.364779 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.364823 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:38:48.372572 16176 raft_consensus.cc:1261] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 1 FOLLOWER]: Refusing update from remote peer d7ee7e67e149461683cdbf8ac50d302e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:48.372572 15914 raft_consensus.cc:1261] T 1fba7b6a97b749ff94875f25c59d45a6 P 46a85e16f7cc4b28ad6beb683b81b139 [term 1 FOLLOWER]: Refusing update from remote peer d7ee7e67e149461683cdbf8ac50d302e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:48.373634 16246 consensus_queue.cc:1035] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46a85e16f7cc4b28ad6beb683b81b139" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:48.373940 16249 consensus_queue.cc:1035] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4892b3ea7224daeb606964f6a058980" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:48.382424 16254 mvcc.cc:205] Tried to move back new op lower bound from 7333586241003954176 to 7333586240845557760. Current Snapshot: MvccSnapshot[applied={T|T < 7333586241003954176}]
I20260926 12:38:48.384670 16253 mvcc.cc:205] Tried to move back new op lower bound from 7333586241003954176 to 7333586240845557760. Current Snapshot: MvccSnapshot[applied={T|T < 7333586241003954176}]
I20260926 12:38:48.392259 15771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:48.392491 15771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:48.395485 15771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e88c80c39704a5d96971505d9815cc9: Sending DeleteTablet for 3 replicas of tablet 1fba7b6a97b749ff94875f25c59d45a6
I20260926 12:38:48.396587 15894 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fba7b6a97b749ff94875f25c59d45a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:48 UTC) from {username='slave'} at 127.0.0.1:48966
I20260926 12:38:48.396582 16025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fba7b6a97b749ff94875f25c59d45a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:48 UTC) from {username='slave'} at 127.0.0.1:48022
I20260926 12:38:48.396672 16156 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fba7b6a97b749ff94875f25c59d45a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:48 UTC) from {username='slave'} at 127.0.0.1:51320
I20260926 12:38:48.397305 16264 tablet_replica.cc:330] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: stopping tablet replica
I20260926 12:38:48.397594 16263 tablet_replica.cc:330] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: stopping tablet replica
I20260926 12:38:48.397732 16264 raft_consensus.cc:2229] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:48.397866 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 15832
I20260926 12:38:48.397939 16263 raft_consensus.cc:2229] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:48.398114 16264 raft_consensus.cc:2258] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:48.398381 16263 raft_consensus.cc:2258] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:48.399997 16264 ts_tablet_manager.cc:1905] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:48.399997 16263 ts_tablet_manager.cc:1905] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:48.402863 16263 ts_tablet_manager.cc:1918] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:48.402861 16264 ts_tablet_manager.cc:1918] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:48.402979 16264 log.cc:1196] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/1fba7b6a97b749ff94875f25c59d45a6
I20260926 12:38:48.402980 16263 log.cc:1196] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/1fba7b6a97b749ff94875f25c59d45a6
I20260926 12:38:48.403311 16263 ts_tablet_manager.cc:1939] T 1fba7b6a97b749ff94875f25c59d45a6 P d7ee7e67e149461683cdbf8ac50d302e: Deleting consensus metadata
I20260926 12:38:48.403311 16264 ts_tablet_manager.cc:1939] T 1fba7b6a97b749ff94875f25c59d45a6 P a4892b3ea7224daeb606964f6a058980: Deleting consensus metadata
I20260926 12:38:48.404850 15758 catalog_manager.cc:4841] TS a4892b3ea7224daeb606964f6a058980 (127.10.224.131:38367): tablet 1fba7b6a97b749ff94875f25c59d45a6 (table test-table [id=eb5ce8a244724385b5bd16cdf7b90f84]) successfully deleted
I20260926 12:38:48.404852 15759 catalog_manager.cc:4841] TS d7ee7e67e149461683cdbf8ac50d302e (127.10.224.130:35047): tablet 1fba7b6a97b749ff94875f25c59d45a6 (table test-table [id=eb5ce8a244724385b5bd16cdf7b90f84]) successfully deleted
W20260926 12:38:48.405925 15758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:42231 (error 108)
I20260926 12:38:48.406050 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 15963
I20260926 12:38:48.408735 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.408328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42231 (local address 127.0.0.1:48972)
0926 12:38:48.408556 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:48.408641 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:42231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 12:38:48.408918 15758 catalog_manager.cc:4584] TS 46a85e16f7cc4b28ad6beb683b81b139 (127.10.224.129:42231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fba7b6a97b749ff94875f25c59d45a6: Network error: Client connection negotiation failed: client connection to 127.10.224.129:42231: connect: Connection refused (error 111)
I20260926 12:38:48.411758 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 16094
I20260926 12:38:48.417732 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 15741
2026-09-26T12:38:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1187 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 12:38:48.440153 11138 mini_oidc.cc:152] Starting JWKS server
I20260926 12:38:48.441903 11138 webserver.cc:460] Webserver started at https://127.0.0.1:43001/ using document root <none> and password file <none>
I20260926 12:38:48.442104 11138 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:38:48.442703 11138 webserver.cc:460] Webserver started at http://127.0.0.1:35737/ using document root <none> and password file <none>
2026-09-26T12:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:48Z Disabled control of system clock
I20260926 12:38:48.453418 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:45469
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:33503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:45469
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/testchainca.cert with env {}
W20260926 12:38:48.570271 16271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:48.570585 16271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:48.570627 16271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:48.574802 16271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:48.574884 16271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:48.574900 16271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:48.574918 16271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:48.574934 16271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:38:48.574951 16271 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:38:48.574965 16271 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43001/jwks.json
I20260926 12:38:48.580693 16271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:45469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:45469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:48.581812 16271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:48.582980 16271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:48.592320 16277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:48.592365 16276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:48.592505 16271 server_base.cc:1034] running on GCE node
W20260926 12:38:48.592320 16279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:48.593420 16271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:48.594696 16271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:48.595968 16271 hybrid_clock.cc:648] HybridClock initialized: now 1790426328595950 us; error 37 us; skew 500 ppm
I20260926 12:38:48.604288 16271 webserver.cc:460] Webserver started at http://127.10.224.190:36709/ using document root <none> and password file <none>
I20260926 12:38:48.605021 16271 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:48.605139 16271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:48.605403 16271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:48.607828 16271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "ebf9405223ef4081857eb39e10bd4fbf"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "e1876308acd81c04d79f884a40c01bfe"
server_key_iv: "dad0fc0e58036bb5bad5a1adf191f403"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.608469 16271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "ebf9405223ef4081857eb39e10bd4fbf"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "e1876308acd81c04d79f884a40c01bfe"
server_key_iv: "dad0fc0e58036bb5bad5a1adf191f403"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.612777 16271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:48.615871 16285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:48.617393 16271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:38:48.617619 16271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "ebf9405223ef4081857eb39e10bd4fbf"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "e1876308acd81c04d79f884a40c01bfe"
server_key_iv: "dad0fc0e58036bb5bad5a1adf191f403"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.617769 16271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:48.630290 16271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:48.631138 16271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:48.631371 16271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:48.638994 16271 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:45469
I20260926 12:38:48.639051 16337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:45469 every 8 connection(s)
I20260926 12:38:48.640123 16271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:48.640453 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 16271
I20260926 12:38:48.640600 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:48.640946 11138 external_mini_cluster.cc:1428] Setting key cbad492286f2362efdb5a2606aea31d4
I20260926 12:38:48.644179 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:48.651193 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf: Bootstrap starting.
I20260926 12:38:48.651499 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.642624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.10.224.190:45469)
0926 12:38:48.643460 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.643478 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.643577 (+ 99us) server_negotiation.cc:407] Connection header received
0926 12:38:48.644314 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.644351 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.644800 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.645136 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.645518 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:48.646533 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:48.647848 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:48.648399 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:48.648534 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:48.649511 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.649517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.649532 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.649980 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.650043 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.650064 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.650147 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.650330 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.650664 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":166,"threads_started":1}
I20260926 12:38:48.653764 16338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:48.654670 16338 log.cc:824] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:48.654927 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:38:48.657080 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf: No bootstrap required, opened a new log
I20260926 12:38:48.661120 16338 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf9405223ef4081857eb39e10bd4fbf" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45469 } }
I20260926 12:38:48.661469 16338 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:48.661568 16338 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf9405223ef4081857eb39e10bd4fbf, State: Initialized, Role: FOLLOWER
I20260926 12:38:48.662053 16338 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf9405223ef4081857eb39e10bd4fbf" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45469 } }
I20260926 12:38:48.662213 16338 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:48.662304 16338 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:48.662436 16338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:48.663499 16338 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf9405223ef4081857eb39e10bd4fbf" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45469 } }
I20260926 12:38:48.664007 16338 leader_election.cc:302] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebf9405223ef4081857eb39e10bd4fbf; no voters:
I20260926 12:38:48.664471 16338 leader_election.cc:288] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:48.664610 16343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:48.664886 16343 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [term 1 LEADER]: Becoming Leader. State: Replica: ebf9405223ef4081857eb39e10bd4fbf, State: Running, Role: LEADER
I20260926 12:38:48.665208 16343 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf9405223ef4081857eb39e10bd4fbf" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45469 } }
I20260926 12:38:48.665999 16338 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:48.667213 16344 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebf9405223ef4081857eb39e10bd4fbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf9405223ef4081857eb39e10bd4fbf" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45469 } } }
I20260926 12:38:48.667373 16344 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:48.667232 16345 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebf9405223ef4081857eb39e10bd4fbf. Latest consensus state: current_term: 1 leader_uuid: "ebf9405223ef4081857eb39e10bd4fbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf9405223ef4081857eb39e10bd4fbf" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45469 } } }
I20260926 12:38:48.667724 16345 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:48.668661 16352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:48.671651 16352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:48.677639 16352 catalog_manager.cc:1347] Generated new cluster ID: b293105c0b83479aaaf43b9c36debcb4
I20260926 12:38:48.677739 16352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:48.687351 16352 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:48.688377 16352 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:48.696794 16352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebf9405223ef4081857eb39e10bd4fbf: Generated new TSK 0
I20260926 12:38:48.697623 16352 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 12:38:48.736997 16368 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 12:38:48.737170 16368 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 12:38:48.737604 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.712692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.10.224.190:45469)
0926 12:38:48.712873 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.712878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.713203 (+ 325us) server_negotiation.cc:407] Connection header received
0926 12:38:48.713552 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.713555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.713610 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.713737 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:38:48.714078 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:48.714638 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:48.715575 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:48.715803 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:48.715917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:48.716346 (+ 429us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:38:48.737086 (+ 20740us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 12:38:48.737483 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 12:38:48.737692 16341 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 12:38:48.739202 16367 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:48.712577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:45469 (local address 127.0.0.1:60302)
0926 12:38:48.713087 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:48.713102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:48.713290 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:48.713756 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:48.713759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:48.713768 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:38:48.713917 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:48.713950 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:48.714762 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:48.714769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:48.715428 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:48.715438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:48.715950 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:48.715955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:48.716053 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:48.716110 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:38:48.737469 (+ 21359us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 12:38:48.737735 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.224.190:45469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":141,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.224.190:45469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 12:38:48.740110 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 16271
2026-09-26T12:38:48Z chronyd exiting
I20260926 12:38:48.778199 11138 test_util.cc:175] -----------------------------------------------
I20260926 12:38:48.778293 11138 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790426298078858-11138-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 12:38:48.779475 11138 mini_oidc.cc:152] Starting JWKS server
I20260926 12:38:48.780452 11138 webserver.cc:460] Webserver started at https://127.0.0.1:39431/ using document root <none> and password file <none>
I20260926 12:38:48.780537 11138 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:38:48.780748 11138 webserver.cc:460] Webserver started at http://127.0.0.1:43831/ using document root <none> and password file <none>
2026-09-26T12:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:48Z Disabled control of system clock
I20260926 12:38:48.790266 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35933
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:35933
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39431/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/testchainca.cert with env {}
W20260926 12:38:48.908619 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:48.908933 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:48.908977 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:48.913120 16374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:48.913193 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:48.913209 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:48.913226 16374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:48.913241 16374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:38:48.913259 16374 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:38:48.913272 16374 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39431/jwks.json
I20260926 12:38:48.918975 16374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:35933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39431/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:48.920137 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:48.921175 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:48.930665 16379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:48.930665 16380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:48.930665 16382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:48.930753 16374 server_base.cc:1034] running on GCE node
I20260926 12:38:48.931680 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:48.932945 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:48.934126 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790426328934089 us; error 57 us; skew 500 ppm
I20260926 12:38:48.936275 16374 webserver.cc:460] Webserver started at http://127.10.224.190:43459/ using document root <none> and password file <none>
I20260926 12:38:48.936920 16374 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:48.936995 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:48.937224 16374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:48.939460 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "333d171e53e547d79b5bbeb6ca6f3664"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "d8244ccf2b995a8029f2782106794d61"
server_key_iv: "6458f81426e6c59bb600c727bc042f58"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.940171 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "333d171e53e547d79b5bbeb6ca6f3664"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "d8244ccf2b995a8029f2782106794d61"
server_key_iv: "6458f81426e6c59bb600c727bc042f58"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.944315 16374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:48.947224 16388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:48.948670 16374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:48.948812 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "333d171e53e547d79b5bbeb6ca6f3664"
format_stamp: "Formatted at 2026-09-26 12:38:48 on dist-test-slave-m3pj"
server_key: "d8244ccf2b995a8029f2782106794d61"
server_key_iv: "6458f81426e6c59bb600c727bc042f58"
server_key_version: "encryptionkey@0"
I20260926 12:38:48.948972 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:48.970427 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:48.971235 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:48.971453 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:48.979382 16374 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:35933
I20260926 12:38:48.979441 16440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:35933 every 8 connection(s)
I20260926 12:38:48.980611 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:48.984485 16441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:48.987329 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 16374
I20260926 12:38:48.987486 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:48.987788 11138 external_mini_cluster.cc:1428] Setting key f20e66e501b370aa03d8520b2c53674b
I20260926 12:38:48.991323 16441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664: Bootstrap starting.
I20260926 12:38:48.994305 16441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:48.995209 16441 log.cc:824] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:48.998041 16441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664: No bootstrap required, opened a new log
I20260926 12:38:48.999454 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:48.989512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.10.224.190:35933)
0926 12:38:48.990283 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:48.990305 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:48.990350 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:38:48.991212 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:48.991237 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:48.991834 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:48.992226 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:48.992729 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:48.994222 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:48.995313 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:48.996010 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:48.996157 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:48.997157 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:48.997164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:48.997183 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:48.997731 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:48.997812 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:48.997839 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:48.997943 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:48.998126 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 12:38:48.998451 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":123,"threads_started":1}
I20260926 12:38:49.002830 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:38:49.002779 16441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333d171e53e547d79b5bbeb6ca6f3664" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35933 } }
I20260926 12:38:49.003093 16441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:49.003160 16441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333d171e53e547d79b5bbeb6ca6f3664, State: Initialized, Role: FOLLOWER
I20260926 12:38:49.003727 16441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333d171e53e547d79b5bbeb6ca6f3664" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35933 } }
I20260926 12:38:49.003906 16441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:49.003980 16441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:49.004112 16441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:49.005245 16441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333d171e53e547d79b5bbeb6ca6f3664" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35933 } }
I20260926 12:38:49.005590 16441 leader_election.cc:302] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 333d171e53e547d79b5bbeb6ca6f3664; no voters:
I20260926 12:38:49.005870 16441 leader_election.cc:288] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:49.006040 16446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:49.006286 16446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [term 1 LEADER]: Becoming Leader. State: Replica: 333d171e53e547d79b5bbeb6ca6f3664, State: Running, Role: LEADER
I20260926 12:38:49.006700 16446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333d171e53e547d79b5bbeb6ca6f3664" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35933 } }
I20260926 12:38:49.007315 16441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:49.008616 16448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333d171e53e547d79b5bbeb6ca6f3664. Latest consensus state: current_term: 1 leader_uuid: "333d171e53e547d79b5bbeb6ca6f3664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333d171e53e547d79b5bbeb6ca6f3664" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35933 } } }
I20260926 12:38:49.008638 16447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "333d171e53e547d79b5bbeb6ca6f3664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333d171e53e547d79b5bbeb6ca6f3664" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35933 } } }
I20260926 12:38:49.008786 16447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:49.008788 16448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:49.009212 16455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:49.012795 16455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:49.019465 16455 catalog_manager.cc:1347] Generated new cluster ID: f3fde5be217640e8baa4fdb57bfedc06
I20260926 12:38:49.019586 16455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:49.048445 16455 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:49.049567 16455 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:49.058586 16455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 333d171e53e547d79b5bbeb6ca6f3664: Generated new TSK 0
I20260926 12:38:49.059438 16455 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 12:38:49.095774 16471 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 12:38:49.095887 16471 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 12:38:49.096715 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:49.079057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.10.224.190:35933)
0926 12:38:49.079212 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:49.079216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:49.079363 (+ 147us) server_negotiation.cc:407] Connection header received
0926 12:38:49.079580 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:49.079584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:49.079649 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:49.079784 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:38:49.080065 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.080914 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.081935 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.082298 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.082406 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:49.082926 (+ 520us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:38:49.096235 (+ 13309us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 12:38:49.096563 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 12:38:49.096798 16444 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 12:38:49.096912 16470 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:49.078917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35933 (local address 127.0.0.1:49344)
0926 12:38:49.079263 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:49.079277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:49.079393 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:49.079816 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:49.079819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:49.079834 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:38:49.079949 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:49.079954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.081054 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.081066 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:49.081799 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:49.081810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.082606 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.082610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:49.082686 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:49.082713 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:38:49.096515 (+ 13802us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 12:38:49.096698 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.224.190:35933: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":141,"threads_started":1}
I20260926 12:38:49.097657 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 16374
2026-09-26T12:38:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (358 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 12:38:49.137456 11138 mini_oidc.cc:152] Starting JWKS server
I20260926 12:38:49.138352 11138 webserver.cc:460] Webserver started at https://127.0.0.1:45215/ using document root <none> and password file <none>
I20260926 12:38:49.138470 11138 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:38:49.138756 11138 webserver.cc:460] Webserver started at http://127.0.0.1:35135/ using document root <none> and password file <none>
2026-09-26T12:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:49Z Disabled control of system clock
I20260926 12:38:49.148151 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39767
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:39767
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45215/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 12:38:49.264266 16477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:49.264603 16477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:49.264714 16477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:49.269137 16477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:49.269241 16477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:49.269271 16477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:49.269315 16477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:49.269357 16477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:38:49.269393 16477 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:38:49.269424 16477 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45215/jwks.json
I20260926 12:38:49.275101 16477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:39767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45215/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:49.276409 16477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:49.277869 16477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:49.286880 16482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:49.286952 16483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:49.286932 16477 server_base.cc:1034] running on GCE node
W20260926 12:38:49.286926 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:49.288023 16477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:49.289310 16477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:49.290532 16477 hybrid_clock.cc:648] HybridClock initialized: now 1790426329290504 us; error 34 us; skew 500 ppm
I20260926 12:38:49.292894 16477 webserver.cc:460] Webserver started at http://127.10.224.190:45007/ using document root <none> and password file <none>
I20260926 12:38:49.293666 16477 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:49.293756 16477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:49.294045 16477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:49.296686 16477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "e4d67d77502348c1a3bad4cd38a9518b"
format_stamp: "Formatted at 2026-09-26 12:38:49 on dist-test-slave-m3pj"
server_key: "52695e408f2aacd699cd87d919c202dc"
server_key_iv: "3fe08d98147924ae1675369f5f850f49"
server_key_version: "encryptionkey@0"
I20260926 12:38:49.297317 16477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "e4d67d77502348c1a3bad4cd38a9518b"
format_stamp: "Formatted at 2026-09-26 12:38:49 on dist-test-slave-m3pj"
server_key: "52695e408f2aacd699cd87d919c202dc"
server_key_iv: "3fe08d98147924ae1675369f5f850f49"
server_key_version: "encryptionkey@0"
I20260926 12:38:49.301268 16477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:49.303894 16491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:49.305105 16477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:38:49.305246 16477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "e4d67d77502348c1a3bad4cd38a9518b"
format_stamp: "Formatted at 2026-09-26 12:38:49 on dist-test-slave-m3pj"
server_key: "52695e408f2aacd699cd87d919c202dc"
server_key_iv: "3fe08d98147924ae1675369f5f850f49"
server_key_version: "encryptionkey@0"
I20260926 12:38:49.305366 16477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:49.326589 16477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:49.327363 16477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:49.327625 16477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:49.334996 16477 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:39767
I20260926 12:38:49.335009 16543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:39767 every 8 connection(s)
I20260926 12:38:49.336068 16477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:49.339418 16544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:49.344388 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 16477
I20260926 12:38:49.344548 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:49.344841 11138 external_mini_cluster.cc:1428] Setting key 7843746aa50086fcb3e7adf333e828f6
I20260926 12:38:49.345487 16544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b: Bootstrap starting.
I20260926 12:38:49.348115 16544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:49.348899 16544 log.cc:824] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:49.350901 16544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b: No bootstrap required, opened a new log
I20260926 12:38:49.354647 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:49.346278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.10.224.190:39767)
0926 12:38:49.346969 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:49.346985 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:49.347022 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:38:49.347766 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:49.347786 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:49.348258 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:49.348615 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:49.349155 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.350142 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.351070 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.351653 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.351834 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:49.352747 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:49.352753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:49.352768 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:49.353207 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:49.353267 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:49.353288 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:49.353415 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:49.353546 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 12:38:49.353841 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
I20260926 12:38:49.354851 16544 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d67d77502348c1a3bad4cd38a9518b" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39767 } }
I20260926 12:38:49.355139 16544 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:49.355218 16544 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d67d77502348c1a3bad4cd38a9518b, State: Initialized, Role: FOLLOWER
I20260926 12:38:49.355732 16544 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d67d77502348c1a3bad4cd38a9518b" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39767 } }
I20260926 12:38:49.355913 16544 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:49.355983 16544 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:49.356079 16544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:49.357349 16544 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d67d77502348c1a3bad4cd38a9518b" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39767 } }
I20260926 12:38:49.357751 16544 leader_election.cc:302] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4d67d77502348c1a3bad4cd38a9518b; no voters:
I20260926 12:38:49.357915 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:38:49.358178 16544 leader_election.cc:288] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:49.358230 16549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:49.358426 16549 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [term 1 LEADER]: Becoming Leader. State: Replica: e4d67d77502348c1a3bad4cd38a9518b, State: Running, Role: LEADER
I20260926 12:38:49.358735 16549 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d67d77502348c1a3bad4cd38a9518b" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39767 } }
I20260926 12:38:49.359604 16544 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:49.361443 16551 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4d67d77502348c1a3bad4cd38a9518b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d67d77502348c1a3bad4cd38a9518b" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39767 } } }
I20260926 12:38:49.361603 16551 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:49.361960 16563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:49.361975 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4d67d77502348c1a3bad4cd38a9518b. Latest consensus state: current_term: 1 leader_uuid: "e4d67d77502348c1a3bad4cd38a9518b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d67d77502348c1a3bad4cd38a9518b" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39767 } } }
I20260926 12:38:49.362795 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:49.365361 16563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:49.371780 16563 catalog_manager.cc:1347] Generated new cluster ID: c1e73eae69a44a3e97b6cd150ad17564
I20260926 12:38:49.371871 16563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:49.378865 16563 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:49.380193 16563 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:49.392433 16563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4d67d77502348c1a3bad4cd38a9518b: Generated new TSK 0
I20260926 12:38:49.393330 16563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:38:49.407286 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:49.402043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.10.224.190:39767)
0926 12:38:49.402187 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:49.402192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:49.402409 (+ 217us) server_negotiation.cc:407] Connection header received
0926 12:38:49.402743 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:49.402746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:49.402800 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:49.402922 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:38:49.403189 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.403808 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.404768 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.405000 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.405116 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:49.405471 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:38:49.406856 (+ 1385us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 12:38:49.407129 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 12:38:49.407374 16547 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 12:38:49.407658 16573 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:49.401954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:39767 (local address 127.0.0.1:48466)
0926 12:38:49.402319 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:49.402331 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:49.402462 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:49.402924 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:49.402927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:49.402937 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:38:49.403072 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:49.403078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.403947 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.403951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:49.404639 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:49.404647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.405129 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.405133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:49.405244 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:49.405268 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:38:49.407034 (+ 1766us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 12:38:49.407181 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.224.190:39767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45215/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260926 12:38:49.408269 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 16477
E20260926 12:38:49.420607 16574 webserver.cc:533] Webserver: sslize failed:
E20260926 12:38:49.420696 16574 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T12:38:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T12:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:49Z Disabled control of system clock
I20260926 12:38:49.457623 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:46699
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:46699
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/testchainca.cert with env {}
W20260926 12:38:49.578434 16578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:49.578763 16578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:49.578805 16578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:49.583190 16578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:49.583281 16578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:49.583298 16578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:49.583316 16578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:49.583384 16578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:38:49.583415 16578 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:38:49.583442 16578 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 12:38:49.589475 16578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:46699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:49.590773 16578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:49.591941 16578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:49.602155 16583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:49.602187 16584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:49.602152 16586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:49.602253 16578 server_base.cc:1034] running on GCE node
I20260926 12:38:49.603102 16578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:49.604471 16578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:49.605706 16578 hybrid_clock.cc:648] HybridClock initialized: now 1790426329605681 us; error 41 us; skew 500 ppm
I20260926 12:38:49.607928 16578 webserver.cc:460] Webserver started at http://127.10.224.190:44953/ using document root <none> and password file <none>
I20260926 12:38:49.608557 16578 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:49.608625 16578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:49.608858 16578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:49.611078 16578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "b8c433332483412d81c4a727f621a5c7"
format_stamp: "Formatted at 2026-09-26 12:38:49 on dist-test-slave-m3pj"
server_key: "574014f2ebdc4eccec7e7767b70bf267"
server_key_iv: "3d19da1640a1b7c2d7569990b8f98311"
server_key_version: "encryptionkey@0"
I20260926 12:38:49.611742 16578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "b8c433332483412d81c4a727f621a5c7"
format_stamp: "Formatted at 2026-09-26 12:38:49 on dist-test-slave-m3pj"
server_key: "574014f2ebdc4eccec7e7767b70bf267"
server_key_iv: "3d19da1640a1b7c2d7569990b8f98311"
server_key_version: "encryptionkey@0"
I20260926 12:38:49.615964 16578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:38:49.618722 16592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:49.620426 16578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:49.620607 16578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "b8c433332483412d81c4a727f621a5c7"
format_stamp: "Formatted at 2026-09-26 12:38:49 on dist-test-slave-m3pj"
server_key: "574014f2ebdc4eccec7e7767b70bf267"
server_key_iv: "3d19da1640a1b7c2d7569990b8f98311"
server_key_version: "encryptionkey@0"
I20260926 12:38:49.620751 16578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:49.643278 16578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:49.644433 16578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:49.644670 16578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:49.652583 16578 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:46699
I20260926 12:38:49.652590 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:46699 every 8 connection(s)
I20260926 12:38:49.653705 16578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:49.654801 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 16578
I20260926 12:38:49.654933 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:49.655278 11138 external_mini_cluster.cc:1428] Setting key 7d6a3ed8c1f664e6c6545d4d9d21d84d
I20260926 12:38:49.657570 16645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:49.663880 16645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7: Bootstrap starting.
I20260926 12:38:49.665521 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:49.656824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.10.224.190:46699)
0926 12:38:49.657602 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:49.657620 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:49.657661 (+ 41us) server_negotiation.cc:407] Connection header received
0926 12:38:49.658424 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:49.658443 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:49.658906 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:49.659300 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:49.659810 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.660886 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.661888 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.662400 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.662581 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:49.663633 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:49.663639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:49.663654 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:38:49.664129 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:49.664200 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:38:49.664221 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:49.664333 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:49.664457 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 12:38:49.664753 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":148,"threads_started":1}
I20260926 12:38:49.666478 16645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:49.667316 16645 log.cc:824] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:49.668861 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:38:49.669750 16645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7: No bootstrap required, opened a new log
I20260926 12:38:49.673883 16645 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c433332483412d81c4a727f621a5c7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46699 } }
I20260926 12:38:49.674185 16645 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:49.674264 16645 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c433332483412d81c4a727f621a5c7, State: Initialized, Role: FOLLOWER
I20260926 12:38:49.674731 16645 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c433332483412d81c4a727f621a5c7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46699 } }
I20260926 12:38:49.674889 16645 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:49.674979 16645 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:49.675098 16645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:49.676201 16645 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c433332483412d81c4a727f621a5c7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46699 } }
I20260926 12:38:49.676584 16645 leader_election.cc:302] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c433332483412d81c4a727f621a5c7; no voters:
I20260926 12:38:49.676946 16645 leader_election.cc:288] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:49.677098 16650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:49.677443 16650 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [term 1 LEADER]: Becoming Leader. State: Replica: b8c433332483412d81c4a727f621a5c7, State: Running, Role: LEADER
I20260926 12:38:49.677790 16650 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c433332483412d81c4a727f621a5c7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46699 } }
I20260926 12:38:49.678406 16645 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:49.679862 16652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c433332483412d81c4a727f621a5c7. Latest consensus state: current_term: 1 leader_uuid: "b8c433332483412d81c4a727f621a5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c433332483412d81c4a727f621a5c7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46699 } } }
I20260926 12:38:49.680035 16652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:49.680474 16651 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8c433332483412d81c4a727f621a5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c433332483412d81c4a727f621a5c7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 46699 } } }
I20260926 12:38:49.680603 16651 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7 [sys.catalog]: This master's current role is: LEADER
W20260926 12:38:49.684029 16665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:49.684130 16665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:49.684223 16659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:49.685042 16659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:49.691886 16659 catalog_manager.cc:1347] Generated new cluster ID: 10387afe11df447a8d237b6b08ce29c6
I20260926 12:38:49.691992 16659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:49.721503 16659 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:49.722754 16659 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:49.742022 16659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c433332483412d81c4a727f621a5c7: Generated new TSK 0
I20260926 12:38:49.742862 16659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:38:49.785642 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:49.756273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.10.224.190:46699)
0926 12:38:49.756427 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:49.756433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:49.756622 (+ 189us) server_negotiation.cc:407] Connection header received
0926 12:38:49.756813 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:49.756816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:49.756873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:49.756998 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:38:49.757288 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.757790 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.759086 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.759303 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.759381 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:49.759781 (+ 400us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:38:49.785152 (+ 25371us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 12:38:49.785484 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46790: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 12:38:49.785724 16648 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46790: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 12:38:49.785866 16674 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:49.756106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:46699 (local address 127.0.0.1:46790)
0926 12:38:49.756479 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:49.756494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:49.756630 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:49.757035 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:49.757039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:49.757049 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:38:49.757171 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:49.757179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.757925 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.757929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:49.758953 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:49.758965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:49.759427 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:49.759432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:49.759567 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:49.759601 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:38:49.785413 (+ 25812us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 12:38:49.785618 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.224.190:46699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":141,"threads_started":1}
I20260926 12:38:49.786486 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 16578
2026-09-26T12:38:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (357 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (129 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (132 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:50 dist-test-slave-m3pj krb5kdc[16686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:50 dist-test-slave-m3pj krb5kdc[16686](info): set up 2 sockets
Sep 26 12:38:50 dist-test-slave-m3pj krb5kdc[16686](info): commencing operation
krb5kdc: starting...
W20260926 12:38:51.108920 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:51 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:38:51.247424 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:33467
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:33467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:51.369956 16699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:51.370325 16699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:51.370432 16699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:51.374841 16699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:51.374963 16699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:51.375000 16699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:51.375072 16699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:51.375111 16699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:51.381034 16699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:33467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:33467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:51.382301 16699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:51.383481 16699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:51.393461 16705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:51.393466 16704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:51.393461 16707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:51.393712 16699 server_base.cc:1034] running on GCE node
I20260926 12:38:51.394411 16699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:51.395794 16699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:51.397025 16699 hybrid_clock.cc:648] HybridClock initialized: now 1790426331396960 us; error 85 us; skew 500 ppm
Sep 26 12:38:51 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:51.400135 16699 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:38:51.401173 16699 webserver.cc:460] Webserver started at http://127.10.224.190:42469/ using document root <none> and password file <none>
I20260926 12:38:51.401799 16699 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:51.401902 16699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:51.402153 16699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:51.404466 16699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "dccf345ca11e4bdabc32f5cdbebf519c"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "cbedebc099ade06f99b0fae780b92ed7"
server_key_iv: "35377f82863eabecfe1aa8cfe259d3c8"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.405068 16699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "dccf345ca11e4bdabc32f5cdbebf519c"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "cbedebc099ade06f99b0fae780b92ed7"
server_key_iv: "35377f82863eabecfe1aa8cfe259d3c8"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.409240 16699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:51.412243 16714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:51.413842 16699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:38:51.414028 16699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "dccf345ca11e4bdabc32f5cdbebf519c"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "cbedebc099ade06f99b0fae780b92ed7"
server_key_iv: "35377f82863eabecfe1aa8cfe259d3c8"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.414179 16699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:51.428563 16699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:51.432219 16699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:51.432444 16699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:51.440156 16699 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:33467
I20260926 12:38:51.440181 16766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:33467 every 8 connection(s)
I20260926 12:38:51.441236 16699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:51.444583 16767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:51.445631 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 16699
I20260926 12:38:51.445765 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:51.446121 11138 external_mini_cluster.cc:1428] Setting key e1c7c1eab387ca45b39ad0cdaa9304fd
I20260926 12:38:51.451881 16767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c: Bootstrap starting.
I20260926 12:38:51.454238 16767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:51.455267 16767 log.cc:824] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:38:51 dist-test-slave-m3pj krb5kdc[16686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:51.457717 16767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c: No bootstrap required, opened a new log
I20260926 12:38:51.461491 16767 raft_consensus.cc:348] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccf345ca11e4bdabc32f5cdbebf519c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33467 } }
I20260926 12:38:51.461858 16767 raft_consensus.cc:374] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:51.461949 16767 raft_consensus.cc:729] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccf345ca11e4bdabc32f5cdbebf519c, State: Initialized, Role: FOLLOWER
I20260926 12:38:51.462502 16767 consensus_queue.cc:260] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccf345ca11e4bdabc32f5cdbebf519c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33467 } }
I20260926 12:38:51.462666 16767 raft_consensus.cc:388] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:51.462738 16767 raft_consensus.cc:482] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:51.462843 16767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:51.462872 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:51.447912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.10.224.190:33467)
0926 12:38:51.448458 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:51.448476 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:51.448519 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:38:51.449272 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:51.449300 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:51.449694 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:51.450123 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:51.451128 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.452273 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.453144 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.453647 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.453830 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:51.457105 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:51.457112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:51.457129 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:51.457164 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:51.459272 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:51.459925 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:51.459931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:51.459936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:51.460066 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:51.460410 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:51.460419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:51.460421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:51.460907 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:51.461157 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:51.461469 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 12:38:51.461801 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":144,"threads_started":1}
I20260926 12:38:51.464231 16767 raft_consensus.cc:504] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccf345ca11e4bdabc32f5cdbebf519c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33467 } }
I20260926 12:38:51.464601 16767 leader_election.cc:302] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dccf345ca11e4bdabc32f5cdbebf519c; no voters:
I20260926 12:38:51.464931 16767 leader_election.cc:288] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:51.465046 16772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:51.465262 16772 raft_consensus.cc:686] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [term 1 LEADER]: Becoming Leader. State: Replica: dccf345ca11e4bdabc32f5cdbebf519c, State: Running, Role: LEADER
I20260926 12:38:51.465651 16772 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccf345ca11e4bdabc32f5cdbebf519c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33467 } }
I20260926 12:38:51.466289 16767 sys_catalog.cc:564] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:51.467321 16773 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dccf345ca11e4bdabc32f5cdbebf519c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccf345ca11e4bdabc32f5cdbebf519c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33467 } } }
I20260926 12:38:51.467443 16773 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:51.466846 16774 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccf345ca11e4bdabc32f5cdbebf519c. Latest consensus state: current_term: 1 leader_uuid: "dccf345ca11e4bdabc32f5cdbebf519c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccf345ca11e4bdabc32f5cdbebf519c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33467 } } }
I20260926 12:38:51.467941 16774 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:51.468837 16781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:51.471722 16781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:51.477285 16781 catalog_manager.cc:1347] Generated new cluster ID: cdba561981c64705a4f37e0941bd479d
I20260926 12:38:51.477384 16781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:51.489192 16781 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:51.490365 16781 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:51.494364 16781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dccf345ca11e4bdabc32f5cdbebf519c: Generated new TSK 0
I20260926 12:38:51.495121 16781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:38:51.528088 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33467
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:51.650020 16793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:51.650336 16793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:51.650382 16793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:51.655012 16793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:51.655095 16793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:51.655161 16793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:51.661057 16793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:51.662266 16793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:51.663359 16793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:51.675139 16798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:51.675163 16801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:51.675170 16799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:51.675436 16793 server_base.cc:1034] running on GCE node
I20260926 12:38:51.676051 16793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:51.676857 16793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:51.678095 16793 hybrid_clock.cc:648] HybridClock initialized: now 1790426331678079 us; error 32 us; skew 500 ppm
Sep 26 12:38:51 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:51.681401 16793 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:38:51.682672 16793 webserver.cc:460] Webserver started at http://127.10.224.129:43823/ using document root <none> and password file <none>
I20260926 12:38:51.683333 16793 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:51.683440 16793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:51.683745 16793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:51.686054 16793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "dab41ca6e9954c309327f9fab4202c7f"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "10d31587c51dada817c5ff7ec65bf427"
server_key_iv: "258ad195701363c3983ed5356f3fdfe9"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.686774 16793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "dab41ca6e9954c309327f9fab4202c7f"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "10d31587c51dada817c5ff7ec65bf427"
server_key_iv: "258ad195701363c3983ed5356f3fdfe9"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.691252 16793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:51.694271 16808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:51.696012 16793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:51.696197 16793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "dab41ca6e9954c309327f9fab4202c7f"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "10d31587c51dada817c5ff7ec65bf427"
server_key_iv: "258ad195701363c3983ed5356f3fdfe9"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.696342 16793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:51.720719 16793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:51.724466 16793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:51.724699 16793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:51.725409 16793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:51.726315 16793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:51.726399 16793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:51.726476 16793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:51.726528 16793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:51.736492 16793 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:39183
I20260926 12:38:51.736510 16921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:39183 every 8 connection(s)
I20260926 12:38:51.737520 16793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:51.745782 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 16793
I20260926 12:38:51.745909 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:51.746264 11138 external_mini_cluster.cc:1428] Setting key 3af93fadef3787823defd554ec71de0d
Sep 26 12:38:51 dist-test-slave-m3pj krb5kdc[16686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:51.754854 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:51.739738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:57581 (local address 127.10.224.190:33467)
0926 12:38:51.739927 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:51.739931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:51.741209 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 12:38:51.742774 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:51.742778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:51.742836 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:51.742926 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:51.744548 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.745105 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.746559 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.746922 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.747051 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:51.750318 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:51.750321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:51.750323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:51.750352 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:51.752100 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:51.752720 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:51.752723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:51.752724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:51.752803 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:51.753302 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:51.753305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:51.753306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:51.753591 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:51.753691 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:51.754588 (+ 897us) server_negotiation.cc:299] Negotiation successful
0926 12:38:51.754721 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 12:38:51.756234 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:51.740228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33467 (local address 127.10.224.129:57581)
0926 12:38:51.740823 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:51.740894 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:51.742485 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:51.743149 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:51.743157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:51.743768 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:51.744338 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:51.744368 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.745234 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.745236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:51.746405 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:51.746418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.747115 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.747120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:51.747320 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:51.748098 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:51.748119 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:51.750087 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:51.752249 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:51.752257 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:51.752277 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:51.752588 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:51.752903 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:51.752907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:51.752909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:51.753146 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:51.753724 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:51.753735 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:51.754190 (+ 455us) client_negotiation.cc:769] Sending connection context
0926 12:38:51.754485 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 12:38:51.754924 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:38:51.757808 16922 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33467
I20260926 12:38:51.758088 16922 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:51.758735 16922 heartbeater.cc:511] Master 127.10.224.190:33467 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:38:51.760686 16731 ts_manager.cc:194] Registered new tserver with Master: dab41ca6e9954c309327f9fab4202c7f (127.10.224.129:39183)
I20260926 12:38:51.762467 16731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:57581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:38:51.779340 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33467
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:51.901273 16927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:51.901599 16927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:51.901645 16927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:51.905978 16927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:51.906059 16927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:51.906124 16927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:51.911979 16927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:51.913137 16927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:51.914197 16927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:51.925457 16935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:51.925460 16932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:51.925558 16927 server_base.cc:1034] running on GCE node
W20260926 12:38:51.925457 16933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:51.926378 16927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:51.927256 16927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:51.928508 16927 hybrid_clock.cc:648] HybridClock initialized: now 1790426331928470 us; error 79 us; skew 500 ppm
Sep 26 12:38:51 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:51.931890 16927 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:38:51.933094 16927 webserver.cc:460] Webserver started at http://127.10.224.130:44867/ using document root <none> and password file <none>
I20260926 12:38:51.933795 16927 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:51.933909 16927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:51.934185 16927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:51.936947 16927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "adfd7ec87e9848ed8e06fb2e1375d8d9"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "85c8f5212697d01c4aa8280a6591e34d"
server_key_iv: "7e34cb9c94fd50f1df498114c3e83e03"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.937681 16927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "adfd7ec87e9848ed8e06fb2e1375d8d9"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "85c8f5212697d01c4aa8280a6591e34d"
server_key_iv: "7e34cb9c94fd50f1df498114c3e83e03"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.942291 16927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:51.945375 16942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:51.946933 16927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:38:51.947101 16927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "adfd7ec87e9848ed8e06fb2e1375d8d9"
format_stamp: "Formatted at 2026-09-26 12:38:51 on dist-test-slave-m3pj"
server_key: "85c8f5212697d01c4aa8280a6591e34d"
server_key_iv: "7e34cb9c94fd50f1df498114c3e83e03"
server_key_version: "encryptionkey@0"
I20260926 12:38:51.947244 16927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:51.972074 16927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:51.975991 16927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:51.976226 16927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:51.977015 16927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:51.977973 16927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:51.978048 16927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:51.978123 16927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:51.978175 16927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:51.988101 16927 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:43089
I20260926 12:38:51.988116 17055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:43089 every 8 connection(s)
I20260926 12:38:51.989318 16927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:51.997716 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 16927
I20260926 12:38:51.997848 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:51.998155 11138 external_mini_cluster.cc:1428] Setting key afe2df0b0cbdfa36608202204fbbc967
Sep 26 12:38:51 dist-test-slave-m3pj krb5kdc[16686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426331, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:52.007831 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:51.991434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:45289 (local address 127.10.224.190:33467)
0926 12:38:51.991626 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:51.991631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:51.992587 (+ 956us) server_negotiation.cc:407] Connection header received
0926 12:38:51.993987 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:51.993990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:51.994088 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:51.994209 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:51.996053 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.996611 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.998611 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.998944 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.999110 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:52.002988 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:52.002994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:52.002999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:52.003039 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:52.005119 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.005804 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.005812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.005816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.005930 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.006365 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.006371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.006376 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.006689 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:52.006807 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:52.007414 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 12:38:52.007609 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 12:38:52.008797 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:51.991889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33467 (local address 127.10.224.130:45289)
0926 12:38:51.992402 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:51.992452 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:51.993694 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:51.994642 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:51.994652 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:51.995119 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:51.995839 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:51.995861 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.996778 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.996781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:51.998407 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:51.998424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:51.999188 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:51.999191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:51.999439 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:52.000501 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:52.000531 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:52.002720 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:52.005297 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:52.005304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:52.005321 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:52.005653 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:52.006068 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:52.006071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:52.006073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:52.006243 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:52.006817 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:52.006824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:52.007115 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 12:38:52.007286 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 12:38:52.007744 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:38:52.010306 17056 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33467
I20260926 12:38:52.010643 17056 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:52.011313 17056 heartbeater.cc:511] Master 127.10.224.190:33467 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:38:52.012811 16731 ts_manager.cc:194] Registered new tserver with Master: adfd7ec87e9848ed8e06fb2e1375d8d9 (127.10.224.130:43089)
I20260926 12:38:52.013495 16731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:45289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:38:52.031912 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33467
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:52.156193 17061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:52.156524 17061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:52.156570 17061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:52.160928 17061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:52.161012 17061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:52.161077 17061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:52.166867 17061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:52.168356 17061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:52.169521 17061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:52.180132 17067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:52.180115 17069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:52.180115 17066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:52.180519 17061 server_base.cc:1034] running on GCE node
I20260926 12:38:52.180994 17061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:52.181777 17061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:52.183018 17061 hybrid_clock.cc:648] HybridClock initialized: now 1790426332182957 us; error 47 us; skew 500 ppm
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:52.186523 17061 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:38:52.187767 17061 webserver.cc:460] Webserver started at http://127.10.224.131:36193/ using document root <none> and password file <none>
I20260926 12:38:52.188405 17061 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:52.188546 17061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:52.188750 17061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:52.191002 17061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "4e1372eae9134c439d088cfa0fe7527f"
format_stamp: "Formatted at 2026-09-26 12:38:52 on dist-test-slave-m3pj"
server_key: "e92f52b60ff658daf922926be8e90d19"
server_key_iv: "3a23fc72e990579065f73788b8bfca73"
server_key_version: "encryptionkey@0"
I20260926 12:38:52.191679 17061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "4e1372eae9134c439d088cfa0fe7527f"
format_stamp: "Formatted at 2026-09-26 12:38:52 on dist-test-slave-m3pj"
server_key: "e92f52b60ff658daf922926be8e90d19"
server_key_iv: "3a23fc72e990579065f73788b8bfca73"
server_key_version: "encryptionkey@0"
I20260926 12:38:52.196144 17061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:52.199088 17076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:52.200888 17061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:52.201066 17061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "4e1372eae9134c439d088cfa0fe7527f"
format_stamp: "Formatted at 2026-09-26 12:38:52 on dist-test-slave-m3pj"
server_key: "e92f52b60ff658daf922926be8e90d19"
server_key_iv: "3a23fc72e990579065f73788b8bfca73"
server_key_version: "encryptionkey@0"
I20260926 12:38:52.201211 17061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:52.218575 17061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:52.222343 17061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:52.222605 17061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:52.223321 17061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:52.224320 17061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:52.224397 17061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:52.224470 17061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:52.224519 17061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:52.234228 17061 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:46009
I20260926 12:38:52.234247 17189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:46009 every 8 connection(s)
I20260926 12:38:52.235239 17061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:52.239601 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 17061
I20260926 12:38:52.239735 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:52.240075 11138 external_mini_cluster.cc:1428] Setting key c305789c25dc72f0d308b841c2c32733
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:38:52.250910 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:52.237384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:58009 (local address 127.10.224.190:33467)
0926 12:38:52.237562 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:52.237567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:52.238465 (+ 898us) server_negotiation.cc:407] Connection header received
0926 12:38:52.240096 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:52.240100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:52.240170 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:52.240278 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:52.241811 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.242562 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.243830 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.244061 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.244145 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:52.247129 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:52.247132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:52.247135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:52.247163 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:52.248800 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.249380 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.249383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.249384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.249464 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.249785 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.249787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.249788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.249976 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:52.250054 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:52.250616 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 12:38:52.250736 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 12:38:52.252103 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:52.237755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33467 (local address 127.10.224.131:58009)
0926 12:38:52.238296 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:52.238342 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:52.239835 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:52.240547 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:52.240555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:52.241044 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:52.241623 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:52.241645 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.242681 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.242684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:52.243715 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:52.243725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.244231 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.244234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:52.244476 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:52.245070 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:52.245096 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:52.246920 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:52.248940 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:52.248947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:52.248963 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:52.249274 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:52.249547 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:52.249549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:52.249550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:52.249702 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:52.250069 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:52.250075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:52.250358 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 12:38:52.250561 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 12:38:52.251040 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":117,"threads_started":1}
I20260926 12:38:52.253554 17190 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33467
I20260926 12:38:52.253846 17190 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:52.254500 17190 heartbeater.cc:511] Master 127.10.224.190:33467 requested a full tablet report, sending...
I20260926 12:38:52.255589 16731 ts_manager.cc:194] Registered new tserver with Master: 4e1372eae9134c439d088cfa0fe7527f (127.10.224.131:46009)
I20260926 12:38:52.256172 16731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:58009
I20260926 12:38:52.266688 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/test-admin.keytab.
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:52.286424 11138 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 12:38:52.289702 17197 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:52.288042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.10.224.129:39183)
0926 12:38:52.288456 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:52.288463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:52.288490 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:38:52.288545 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:52.288551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:52.288784 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:52.288935 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:52.289497 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45826: BlockingRecv error: recv got EOF from 127.0.0.1:45826 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":129,"threads_started":1}
W20260926 12:38:52.289863 11138 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5cc' of type FILE
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:52.291121 11138 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:52.292199 11138 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:52.304484 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:52.293463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.10.224.129:39183)
0926 12:38:52.293605 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:52.293608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:52.293623 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:52.293781 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:52.293785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:52.293838 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:52.293949 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:52.294601 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.295499 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.296546 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.296997 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.297096 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:52.300060 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:52.300075 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:52.300094 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:52.300128 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:52.302192 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.302839 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.302846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.302848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.302970 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.303274 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.303277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.303278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.303723 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:52.303923 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:52.304112 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 12:38:52.304262 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 12:38:52.310588 17197 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:52.309236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.10.224.129:39183)
0926 12:38:52.309353 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:52.309358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:52.309508 (+ 150us) server_negotiation.cc:407] Connection header received
0926 12:38:52.309681 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:52.309684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:52.309742 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:52.309831 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:52.310440 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45836: BlockingRecv error: recv got EOF from 127.0.0.1:45836 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 12:38:52.310757 11138 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5cc' of type FILE
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:52.311968 11138 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:52.313141 11138 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:52.323599 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:52.314265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.10.224.129:39183)
0926 12:38:52.314430 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:52.314436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:52.314608 (+ 172us) server_negotiation.cc:407] Connection header received
0926 12:38:52.314808 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:52.314811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:52.314855 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:52.314978 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:52.315680 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.316257 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.317173 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.317436 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.317556 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:52.320163 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:52.320167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:52.320170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:52.320208 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:52.321734 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.322260 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.322265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.322266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.322370 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.322667 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.322672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.322674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.322872 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:52.323047 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:52.323286 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 12:38:52.323412 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 12:38:52.326689 17197 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:52.325620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.10.224.129:39183)
0926 12:38:52.325774 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:52.325778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:52.325836 (+ 58us) server_negotiation.cc:407] Connection header received
0926 12:38:52.325979 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:52.325982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:52.326024 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:52.326106 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:52.326567 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45856: BlockingRecv error: recv got EOF from 127.0.0.1:45856 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 12:38:52.326836 11138 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790426298078858-11138-0/krb5kdc/krb5cc' of type FILE
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:52.328265 11138 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:52.329293 11138 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[16686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:38:52.339401 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:52.330377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.10.224.129:39183)
0926 12:38:52.330525 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:52.330531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:52.330625 (+ 94us) server_negotiation.cc:407] Connection header received
0926 12:38:52.330825 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:52.330828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:52.330875 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:52.330977 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:52.331553 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.332074 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.333125 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:52.333358 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:52.333490 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:52.336226 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:52.336230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:52.336232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:52.336264 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:52.337661 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.338151 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.338155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.338157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.338240 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:52.338526 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:52.338531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:52.338533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:52.338715 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:52.338894 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:52.339124 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 12:38:52.339236 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:38:52.340687 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 16793
I20260926 12:38:52.347162 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 16927
I20260926 12:38:52.353502 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 17061
I20260926 12:38:52.359475 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 16699
2026-09-26T12:38:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2314 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[17210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[17210](info): set up 2 sockets
Sep 26 12:38:52 dist-test-slave-m3pj krb5kdc[17210](info): commencing operation
krb5kdc: starting...
W20260926 12:38:53.440912 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:38:53 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.190.keytab.
Entry for principal oryx/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.190.keytab.
I20260926 12:38:53.579443 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38019
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.190.keytab
--principal=oryx/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:38019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:53.695256 17223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:53.695641 17223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:53.695731 17223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:53.699895 17223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:53.699995 17223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:53.700034 17223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:53.700071 17223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:53.700105 17223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:53.705684 17223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:38019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.190.keytab
--principal=oryx/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:53.706943 17223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:53.708062 17223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:53.716583 17228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:53.716588 17229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:53.716624 17231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:53.716956 17223 server_base.cc:1034] running on GCE node
I20260926 12:38:53.717512 17223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:53.718668 17223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:53.719826 17223 hybrid_clock.cc:648] HybridClock initialized: now 1790426333719807 us; error 33 us; skew 500 ppm
Sep 26 12:38:53 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426333, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:53.729341 17223 init.cc:377] Logged in from keytab as oryx/127.10.224.190@KRBTEST.COM (short username oryx)
I20260926 12:38:53.730393 17223 webserver.cc:460] Webserver started at http://127.10.224.190:38191/ using document root <none> and password file <none>
I20260926 12:38:53.730952 17223 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:53.731011 17223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:53.731190 17223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:53.733407 17223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "c19fed68d8e04971bcf0fb5fd081c5e4"
format_stamp: "Formatted at 2026-09-26 12:38:53 on dist-test-slave-m3pj"
server_key: "2c60b087b85b0834a8291bbf6873f5d8"
server_key_iv: "1b55fecab3c2462d5c96fd42a8be3406"
server_key_version: "encryptionkey@0"
I20260926 12:38:53.734133 17223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "c19fed68d8e04971bcf0fb5fd081c5e4"
format_stamp: "Formatted at 2026-09-26 12:38:53 on dist-test-slave-m3pj"
server_key: "2c60b087b85b0834a8291bbf6873f5d8"
server_key_iv: "1b55fecab3c2462d5c96fd42a8be3406"
server_key_version: "encryptionkey@0"
I20260926 12:38:53.738332 17223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:53.741080 17238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:53.742517 17223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:38:53.742655 17223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "c19fed68d8e04971bcf0fb5fd081c5e4"
format_stamp: "Formatted at 2026-09-26 12:38:53 on dist-test-slave-m3pj"
server_key: "2c60b087b85b0834a8291bbf6873f5d8"
server_key_iv: "1b55fecab3c2462d5c96fd42a8be3406"
server_key_version: "encryptionkey@0"
I20260926 12:38:53.742782 17223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:53.759763 17223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:53.763432 17223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:53.763731 17223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:53.771173 17223 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:38019
I20260926 12:38:53.771497 17290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:38019 every 8 connection(s)
I20260926 12:38:53.772295 17223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:53.775177 17291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:53.776574 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 17223
I20260926 12:38:53.776767 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:38:53.777200 11138 external_mini_cluster.cc:1428] Setting key 064a9aad9271221e820331954259dff2
I20260926 12:38:53.782152 17291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: Bootstrap starting.
I20260926 12:38:53.784999 17291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:53.786020 17291 log.cc:824] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:38:53 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:38:53.788492 17291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: No bootstrap required, opened a new log
I20260926 12:38:53.792339 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:53.778769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.10.224.190:38019)
0926 12:38:53.779194 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:53.779211 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:53.779248 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:38:53.779958 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:53.779978 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:53.780333 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:53.780642 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:53.781619 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:53.782715 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:53.783905 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:53.784409 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:53.784548 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:53.787638 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:53.787643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:53.787658 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:53.787685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:53.789371 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:53.789824 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:53.789831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:53.789837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:53.789956 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:53.790231 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:53.790234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:53.790236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:53.790673 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:53.790890 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:53.791180 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 12:38:53.791489 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260926 12:38:53.792244 17291 raft_consensus.cc:348] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } }
I20260926 12:38:53.792522 17291 raft_consensus.cc:374] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:53.792608 17291 raft_consensus.cc:729] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19fed68d8e04971bcf0fb5fd081c5e4, State: Initialized, Role: FOLLOWER
I20260926 12:38:53.793089 17291 consensus_queue.cc:260] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } }
I20260926 12:38:53.793251 17291 raft_consensus.cc:388] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:38:53.793335 17291 raft_consensus.cc:482] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:38:53.793443 17291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:53.794710 17291 raft_consensus.cc:504] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } }
I20260926 12:38:53.795087 17291 leader_election.cc:302] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c19fed68d8e04971bcf0fb5fd081c5e4; no voters:
I20260926 12:38:53.795370 17291 leader_election.cc:288] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:38:53.795583 17296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:53.795816 17296 raft_consensus.cc:686] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 1 LEADER]: Becoming Leader. State: Replica: c19fed68d8e04971bcf0fb5fd081c5e4, State: Running, Role: LEADER
I20260926 12:38:53.796212 17296 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } }
I20260926 12:38:53.796710 17291 sys_catalog.cc:564] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:53.798002 17298 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c19fed68d8e04971bcf0fb5fd081c5e4. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } }
I20260926 12:38:53.798128 17298 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:53.798487 17297 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } }
I20260926 12:38:53.798617 17297 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:53.798552 17305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:53.801674 17305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:53.807834 17305 catalog_manager.cc:1347] Generated new cluster ID: ab7db6a15d1d43bcacdf3cea4b45ead3
I20260926 12:38:53.807947 17305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:53.834364 17305 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:53.835592 17305 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:53.847235 17305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: Generated new TSK 0
I20260926 12:38:53.848173 17305 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab.
Entry for principal oryx/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab.
I20260926 12:38:53.885062 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38019
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab
--principal=oryx/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:54.005695 17317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:54.006026 17317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:54.006073 17317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:54.010233 17317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:54.010388 17317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:54.010458 17317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:54.016245 17317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab
--principal=oryx/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:54.017336 17317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:54.018353 17317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:54.029357 17325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.029341 17323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.029341 17322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:54.029901 17317 server_base.cc:1034] running on GCE node
I20260926 12:38:54.030373 17317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:54.031066 17317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:54.032265 17317 hybrid_clock.cc:648] HybridClock initialized: now 1790426334032221 us; error 58 us; skew 500 ppm
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:54.035588 17317 init.cc:377] Logged in from keytab as oryx/127.10.224.129@KRBTEST.COM (short username oryx)
I20260926 12:38:54.036875 17317 webserver.cc:460] Webserver started at http://127.10.224.129:42339/ using document root <none> and password file <none>
I20260926 12:38:54.037618 17317 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:54.037729 17317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:54.037981 17317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:54.040370 17317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "4631bcece0204ad2a8252fc5fe1a0d94"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "aaee775c5df2b52b5b28a7633804cb9c"
server_key_iv: "fc69a72ad70985b12c80837dc8e7695f"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.041023 17317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "4631bcece0204ad2a8252fc5fe1a0d94"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "aaee775c5df2b52b5b28a7633804cb9c"
server_key_iv: "fc69a72ad70985b12c80837dc8e7695f"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.045553 17317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:54.048635 17332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.050381 17317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:54.050747 17317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "4631bcece0204ad2a8252fc5fe1a0d94"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "aaee775c5df2b52b5b28a7633804cb9c"
server_key_iv: "fc69a72ad70985b12c80837dc8e7695f"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.050935 17317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:54.079491 17317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:54.084066 17317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:54.084307 17317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:54.084980 17317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:54.085973 17317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:54.086035 17317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.086083 17317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:54.086122 17317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.096699 17317 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:44163
I20260926 12:38:54.096717 17445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:44163 every 8 connection(s)
I20260926 12:38:54.098004 17317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:54.103371 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 17317
I20260926 12:38:54.103626 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:38:54.103961 11138 external_mini_cluster.cc:1428] Setting key 80c45d7677d89f0171028d49122ee1b6
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.130@KRBTEST.COM" created.
I20260926 12:38:54.122334 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.101575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:44319 (local address 127.10.224.190:38019)
0926 12:38:54.101879 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.101884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.103317 (+ 1433us) server_negotiation.cc:407] Connection header received
0926 12:38:54.105843 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.105847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.105938 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.106048 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.108401 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.109547 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.111141 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.111659 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.111816 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.115625 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.115630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.115633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.115669 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.118347 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.119682 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.119696 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.119702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.119861 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.120566 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.120569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.120571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.120825 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.120952 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.121889 (+ 937us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.122090 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260926 12:38:54.123124 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.102011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:44319)
0926 12:38:54.102930 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.103017 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.105489 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.106357 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.106371 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.107118 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.108107 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.108142 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.109791 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.109795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.110892 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.110953 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.111890 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.111893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.112087 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.112987 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.113007 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.115283 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.118666 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.118675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.118693 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.119453 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.120055 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.120059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.120061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.120351 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.120976 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.120987 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.121451 (+ 464us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.121709 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.122058 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":205,"threads_started":1}
I20260926 12:38:54.124698 17446 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38019
I20260926 12:38:54.124997 17446 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:54.125696 17446 heartbeater.cc:511] Master 127.10.224.190:38019 requested a full tablet report, sending...
I20260926 12:38:54.127902 17255 ts_manager.cc:194] Registered new tserver with Master: 4631bcece0204ad2a8252fc5fe1a0d94 (127.10.224.129:44163)
I20260926 12:38:54.129206 17255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:44319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab.
Entry for principal oryx/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab.
I20260926 12:38:54.140090 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38019
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab
--principal=oryx/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:54.264276 17451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:54.264606 17451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:54.264655 17451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:54.270097 17451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:54.270201 17451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:54.270311 17451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:54.276840 17451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab
--principal=oryx/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:54.278258 17451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:54.279481 17451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:54.290656 17459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.290645 17456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:54.290707 17451 server_base.cc:1034] running on GCE node
W20260926 12:38:54.290645 17457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:54.291430 17451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:54.292268 17451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:54.293455 17451 hybrid_clock.cc:648] HybridClock initialized: now 1790426334293429 us; error 42 us; skew 500 ppm
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:54.296605 17451 init.cc:377] Logged in from keytab as oryx/127.10.224.130@KRBTEST.COM (short username oryx)
I20260926 12:38:54.297749 17451 webserver.cc:460] Webserver started at http://127.10.224.130:39871/ using document root <none> and password file <none>
I20260926 12:38:54.298457 17451 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:54.298549 17451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:54.298794 17451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:54.301020 17451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "ea61906bb2b54dfab8df4c3cae1b812a"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "6d17b76a6c0f288c57800d3820f697d8"
server_key_iv: "5b17cf5a08eb6a2eba0d329f75d35cb8"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.301589 17451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "ea61906bb2b54dfab8df4c3cae1b812a"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "6d17b76a6c0f288c57800d3820f697d8"
server_key_iv: "5b17cf5a08eb6a2eba0d329f75d35cb8"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.305644 17451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:54.308321 17466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.309765 17451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:38:54.309916 17451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "ea61906bb2b54dfab8df4c3cae1b812a"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "6d17b76a6c0f288c57800d3820f697d8"
server_key_iv: "5b17cf5a08eb6a2eba0d329f75d35cb8"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.310051 17451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:54.334724 17451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:54.338382 17451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:54.338706 17451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:54.339368 17451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:54.340297 17451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:54.340387 17451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.340467 17451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:54.340510 17451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.350776 17451 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:41439
I20260926 12:38:54.350823 17579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:41439 every 8 connection(s)
I20260926 12:38:54.351912 17451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:54.358093 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 17451
I20260926 12:38:54.358232 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:38:54.358551 11138 external_mini_cluster.cc:1428] Setting key 473d9d40462502a67daa27120adcbdf2
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:38:54.370015 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.354101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:42849 (local address 127.10.224.190:38019)
0926 12:38:54.354284 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.354290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.355876 (+ 1586us) server_negotiation.cc:407] Connection header received
0926 12:38:54.357366 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.357369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.357437 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.357520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.359269 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.359873 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.361783 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.362087 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.362221 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.365301 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.365305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.365307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.365335 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.367213 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.368046 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.368050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.368052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.368157 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.368561 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.368564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.368566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.368798 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.368899 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.369594 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.369809 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 12:38:54.370805 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.354743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:42849)
0926 12:38:54.355702 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.355754 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.357118 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.357717 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.357725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.358315 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.359085 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.359105 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.360017 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.360021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.361607 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.361626 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.362278 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.362283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.362482 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.363165 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.363183 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.365078 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.367395 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.367404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.367427 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.367919 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.368263 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.368266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.368269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.368452 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.368943 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.368955 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.369303 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.369490 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.369843 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":198,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.131@KRBTEST.COM" created.
I20260926 12:38:54.372148 17580 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38019
I20260926 12:38:54.372439 17580 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:54.373088 17580 heartbeater.cc:511] Master 127.10.224.190:38019 requested a full tablet report, sending...
I20260926 12:38:54.374550 17255 ts_manager.cc:194] Registered new tserver with Master: ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439)
I20260926 12:38:54.375208 17255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:42849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab.
Entry for principal oryx/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab.
I20260926 12:38:54.391803 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38019
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab
--principal=oryx/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:54.511861 17585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:54.512212 17585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:54.512287 17585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:54.516893 17585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:54.516992 17585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:54.517088 17585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:54.522976 17585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab
--principal=oryx/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:54.524302 17585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:54.525419 17585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:38:54.536293 17585 server_base.cc:1034] running on GCE node
W20260926 12:38:54.536242 17593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.536249 17590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.536249 17591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:54.536998 17585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:54.537770 17585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:54.538946 17585 hybrid_clock.cc:648] HybridClock initialized: now 1790426334538928 us; error 31 us; skew 500 ppm
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:54.542380 17585 init.cc:377] Logged in from keytab as oryx/127.10.224.131@KRBTEST.COM (short username oryx)
I20260926 12:38:54.543397 17585 webserver.cc:460] Webserver started at http://127.10.224.131:34175/ using document root <none> and password file <none>
I20260926 12:38:54.544080 17585 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:54.544155 17585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:54.544368 17585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:54.546758 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "89039a18f4734d29ae5de038215d21aa"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "72c184c8e7fb69adab335cf5b8e619f2"
server_key_iv: "697b300abf284ba6d8588a955e2d41cc"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.547425 17585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "89039a18f4734d29ae5de038215d21aa"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "72c184c8e7fb69adab335cf5b8e619f2"
server_key_iv: "697b300abf284ba6d8588a955e2d41cc"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.551895 17585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:38:54.554663 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.556170 17585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:54.556310 17585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "89039a18f4734d29ae5de038215d21aa"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "72c184c8e7fb69adab335cf5b8e619f2"
server_key_iv: "697b300abf284ba6d8588a955e2d41cc"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.556449 17585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:54.583777 17585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:54.587438 17585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:54.587700 17585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:54.588356 17585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:54.589263 17585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:54.589334 17585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.589412 17585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:54.589462 17585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.603865 17585 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:33013
I20260926 12:38:54.603897 17713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:33013 every 8 connection(s)
I20260926 12:38:54.605310 17585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:54.609733 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 17585
I20260926 12:38:54.609898 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:38:54.610285 11138 external_mini_cluster.cc:1428] Setting key 58ebaee2cdd14387811976df92cc33d8
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:38:54.626346 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.608073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:58233 (local address 127.10.224.190:38019)
0926 12:38:54.608274 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.608280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.610962 (+ 2682us) server_negotiation.cc:407] Connection header received
0926 12:38:54.613016 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.613019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.613070 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.613142 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.614771 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.615453 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.617656 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.617898 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.618019 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.621486 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.621493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.621498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.621534 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.623700 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.624535 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.624542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.624546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.624649 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.625081 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.625086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.625088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.625366 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.625467 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.626043 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.626182 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 12:38:54.627612 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.608648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.131:58233)
0926 12:38:54.610735 (+ 2087us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.610818 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.612755 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.613367 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.613378 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.613988 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.614612 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.614629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.615681 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.615684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.617508 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.617525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.618073 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.618077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.618311 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.619120 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.619138 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.621199 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.623863 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.623872 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.623893 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.624402 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.624789 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.624792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.624793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.624982 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.625477 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.625487 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.625777 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.626004 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.626673 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":260,"threads_started":1}
I20260926 12:38:54.629169 17714 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38019
I20260926 12:38:54.629535 17714 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:54.630354 17714 heartbeater.cc:511] Master 127.10.224.190:38019 requested a full tablet report, sending...
I20260926 12:38:54.631924 17255 ts_manager.cc:194] Registered new tserver with Master: 89039a18f4734d29ae5de038215d21aa (127.10.224.131:33013)
I20260926 12:38:54.632779 17255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:58233
I20260926 12:38:54.637692 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.224.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.189.keytab.
Entry for principal oryx/127.10.224.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.189.keytab.
I20260926 12:38:54.671602 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.189:33525
--webserver_interface=127.10.224.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.189.keytab
--principal=oryx/127.10.224.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:38019,127.10.224.189:33525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:54.789888 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:54.790201 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:54.790243 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:54.794407 17719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:54.794487 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:54.794507 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:54.794524 17719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:54.794540 17719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:54.800516 17719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:38019,127.10.224.189:33525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.189.keytab
--principal=oryx/127.10.224.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.189:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.224.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
W20260926 12:38:54.800738 17719 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.224.190:38019,127.10.224.189:33525'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 12:38:54.801678 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:54.802691 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:54.811367 17727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.811367 17724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.811384 17725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:54.811857 17719 server_base.cc:1034] running on GCE node
I20260926 12:38:54.812476 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:54.813668 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:54.814863 17719 hybrid_clock.cc:648] HybridClock initialized: now 1790426334814845 us; error 33 us; skew 500 ppm
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:54.817934 17719 init.cc:377] Logged in from keytab as oryx/127.10.224.189@KRBTEST.COM (short username oryx)
I20260926 12:38:54.818929 17719 webserver.cc:460] Webserver started at http://127.10.224.189:35633/ using document root <none> and password file <none>
I20260926 12:38:54.819591 17719 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:54.819681 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:54.819908 17719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:54.822132 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data/instance:
uuid: "747538dff7b84fb6864644aacd55ff47"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "2538917fe387c945b59b69c6d04a3278"
server_key_iv: "6351810c6f88a5d66cee0b8b379d7116"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.822748 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal/instance:
uuid: "747538dff7b84fb6864644aacd55ff47"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "2538917fe387c945b59b69c6d04a3278"
server_key_iv: "6351810c6f88a5d66cee0b8b379d7116"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.826903 17719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:54.829555 17734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.830892 17719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:38:54.831058 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
uuid: "747538dff7b84fb6864644aacd55ff47"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "2538917fe387c945b59b69c6d04a3278"
server_key_iv: "6351810c6f88a5d66cee0b8b379d7116"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.831188 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:54.843343 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:54.846675 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:54.846891 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:54.854312 17719 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.189:33525
I20260926 12:38:54.854628 17786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.189:33525 every 8 connection(s)
I20260926 12:38:54.855374 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
I20260926 12:38:54.858563 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 17719
I20260926 12:38:54.858610 17787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:54.864102 17787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.224.189@KRBTEST.COM
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.189@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:38:54.875208 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.860215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.10.224.189:33525)
0926 12:38:54.860664 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.860688 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.860725 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:38:54.861570 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.861597 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.861971 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.862279 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.863275 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.864386 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.865590 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.866266 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.866455 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.869432 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.869438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.869448 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.869475 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.871579 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.872055 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.872062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.872069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.872191 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.872502 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.872506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.872507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.873051 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.873221 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.873656 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.874037 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":184,"threads_started":1}
I20260926 12:38:54.876187 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.866300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34386)
0926 12:38:54.866828 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.866878 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.867037 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.867379 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.867383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.867827 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.867965 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.867971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.868828 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.868832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.869722 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.869731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.870277 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.870281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.870398 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.870927 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.870939 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.872362 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.874327 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.874334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.874340 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.874615 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.874903 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.874906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.874909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.875001 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.875640 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.875646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.875764 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.875887 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.876041 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":123,"threads_started":1}
I20260926 12:38:54.876278 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.866366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.10.224.190:38019)
0926 12:38:54.866573 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.866578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.866950 (+ 372us) server_negotiation.cc:407] Connection header received
0926 12:38:54.867163 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.867168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.867231 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.867329 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.868087 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.868690 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.869831 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.870154 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.870277 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.872510 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.872515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.872517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.872554 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.874219 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.874714 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.874717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.874718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.874813 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.875124 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.875130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.875131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.875496 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.875638 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.876003 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.876131 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260926 12:38:54.877727 17787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.189@KRBTEST.COM for oryx/127.10.224.189@KRBTEST.COM
I20260926 12:38:54.886384 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.878254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.0.0.1:46934)
0926 12:38:54.878430 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.878446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.878548 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.878764 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.878767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.878945 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.879026 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.879031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.880194 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.880197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.880710 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.880716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.881140 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.881143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.881228 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.881753 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.881764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.883055 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.884874 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.884878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.884880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.885068 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.885326 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.885330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.885332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.885415 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.885785 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.885788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.885874 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.885979 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.886177 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":5}
I20260926 12:38:54.886384 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.878305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.10.224.189:33525)
0926 12:38:54.878430 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.878435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.878507 (+ 72us) server_negotiation.cc:407] Connection header received
0926 12:38:54.878637 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.878640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.878678 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.878754 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.879167 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.880065 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.880824 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.881038 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.881140 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.883208 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.883211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.883213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.883231 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.884746 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.885159 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.885165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.885167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.885241 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.885508 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.885511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.885513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.885692 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.885788 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.886021 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.886226 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":49}
I20260926 12:38:54.889633 17787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Bootstrap starting.
I20260926 12:38:54.892115 17787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:54.892894 17787 log.cc:824] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:54.894914 17787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: No bootstrap required, opened a new log
I20260926 12:38:54.898218 17787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } }
I20260926 12:38:54.898437 17787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:54.898517 17787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747538dff7b84fb6864644aacd55ff47, State: Initialized, Role: FOLLOWER
I20260926 12:38:54.899011 17787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } }
I20260926 12:38:54.899576 17793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } } }
I20260926 12:38:54.899724 17793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:54.900329 17787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 12:38:54.905617 17804 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:54.905725 17804 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:54.907215 17719 master_runner.cc:428] Detected that this master 747538dff7b84fb6864644aacd55ff47 is joining an existing cluster
I20260926 12:38:54.907295 17719 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.224.189:33525
I20260926 12:38:54.918709 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.909511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34398)
0926 12:38:54.910265 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.910276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.910369 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.910596 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.910598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.910787 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.910895 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.910900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.912050 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.912054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.913019 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.913031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.913686 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.913690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.913783 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.914384 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.914399 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.914915 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.916869 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.916873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.916875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.917148 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.917466 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.917469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.917479 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.917571 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.918051 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.918057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.918182 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.918310 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.918479 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":103,"threads_started":1}
I20260926 12:38:54.919325 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.909631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.10.224.190:38019)
0926 12:38:54.909763 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.909767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.910355 (+ 588us) server_negotiation.cc:407] Connection header received
0926 12:38:54.910444 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.910448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.910505 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.910592 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.911007 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.911904 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.913141 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.913537 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.913645 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.915120 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.915124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.915127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.915162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.916764 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.917266 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.917270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.917272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.917362 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.917661 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.917664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.917666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.917916 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.918043 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.918398 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.919135 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:38:54.921514 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.910038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.10.224.189:33525)
0926 12:38:54.910147 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.910150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.911838 (+ 1688us) server_negotiation.cc:407] Connection header received
0926 12:38:54.911948 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.911951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.911990 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.912075 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.912480 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.912946 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.913995 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.914348 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.914479 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.917621 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.917626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.917629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.917653 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.919290 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.920350 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.920356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.920358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.920435 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.920690 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.920693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.920695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.920855 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.920958 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.921210 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.921341 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":43}
I20260926 12:38:54.922132 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.909941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.0.0.1:46950)
0926 12:38:54.911749 (+ 1808us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.911762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.911878 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.912098 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.912101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.912288 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.912373 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.912378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.913055 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.913058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.913871 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.913879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.914477 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.914481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.914619 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.917042 (+ 2423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.917054 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.917497 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.919543 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.919546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.919548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.920253 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.920520 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.920524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.920526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.920604 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.920949 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.920953 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.921047 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.921774 (+ 727us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.921929 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1732,"mutex_wait_us":22,"thread_start_us":85,"threads_started":1}
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426333, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.190@KRBTEST.COM for oryx/127.10.224.189@KRBTEST.COM
I20260926 12:38:54.934686 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.924291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.10.224.189:33525)
0926 12:38:54.924418 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.924422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.924791 (+ 369us) server_negotiation.cc:407] Connection header received
0926 12:38:54.924988 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.924991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.925055 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.925122 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.925986 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.926517 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.927867 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.928107 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.928248 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.930845 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.930849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.930852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.930898 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.932668 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.933204 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.933208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.933210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.933290 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.933603 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.933608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.933610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.933880 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.933991 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.934388 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.934516 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:38:54.934731 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.924237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.0.0.1:46962)
0926 12:38:54.924641 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.924705 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.924850 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.925185 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.925192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.925695 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.925839 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.925847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.926620 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.926624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.927714 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.927727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.928259 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.928262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.928460 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.929171 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.929193 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.930699 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.932800 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.932811 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.932819 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.933092 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.933399 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.933402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.933404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.933488 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.933991 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.934003 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.934170 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.934336 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.934574 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
I20260926 12:38:54.936545 17255 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.224.189:33525: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "747538dff7b84fb6864644aacd55ff47"
member_type: NON_VOTER
last_known_addr {
host: "127.10.224.189"
port: 33525
}
attrs {
promote: true
}
}
dest_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4"
cas_config_opid_index: -1
I20260926 12:38:54.937220 17255 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } }
I20260926 12:38:54.939044 17816 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 747538dff7b84fb6864644aacd55ff47 (127.10.224.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } }
I20260926 12:38:54.939936 17761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:54.940660 17816 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.224.189:33525
I20260926 12:38:54.941037 17816 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } }
I20260926 12:38:54.941143 17816 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:54.941146 17719 master.cc:537] Master@127.10.224.189:33525 shutting down...
WARNING: no policy specified for oryx/127.10.224.188@KRBTEST.COM; defaulting to no policy
I20260926 12:38:54.941609 17761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Refusing update from remote peer c19fed68d8e04971bcf0fb5fd081c5e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 12:38:54.942432 17815 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:38:54.944684 17239 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 12:38:54.945631 17239 consensus_peers.cc:487] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 -> Peer 747538dff7b84fb6864644aacd55ff47 (127.10.224.189:33525): Couldn't send request to peer 747538dff7b84fb6864644aacd55ff47. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 12:38:54.946290 17719 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:54.946584 17719 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:54.946702 17719 tablet_replica.cc:330] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.188@KRBTEST.COM" created.
I20260926 12:38:54.951164 17719 master.cc:552] Master@127.10.224.189:33525 shutdown complete.
I20260926 12:38:54.951241 17719 master_runner.cc:305] Clearing existing system tablet
I20260926 12:38:54.951521 17719 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:54.951617 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:54.953282 17719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:54.954303 17821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:54.954602 17719 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:54.954715 17719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:54.954754 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
uuid: "747538dff7b84fb6864644aacd55ff47"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "2538917fe387c945b59b69c6d04a3278"
server_key_iv: "6351810c6f88a5d66cee0b8b379d7116"
server_key_version: "encryptionkey@0"
I20260926 12:38:54.957168 17719 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:54.960573 17719 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 12:38:54.960772 17719 log.cc:1196] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 12:38:54.961083 17719 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Deleting consensus metadata
I20260926 12:38:54.961261 17719 master_runner.cc:315] Copying system tablet from 127.10.224.190:38019
I20260926 12:38:54.962718 17719 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.10.224.190:38019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.188.keytab.
Entry for principal oryx/127.10.224.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.188.keytab.
I20260926 12:38:54.966711 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.188:35243
--webserver_interface=127.10.224.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.188.keytab
--principal=oryx/127.10.224.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:38019,127.10.224.189:33525,127.10.224.188:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:38:54.972685 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.963484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.10.224.190:38019)
0926 12:38:54.963692 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:54.963697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:54.963996 (+ 299us) server_negotiation.cc:407] Connection header received
0926 12:38:54.964102 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:54.964106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:54.964160 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:54.964363 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:54.965040 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.965630 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.967018 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.967419 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.967590 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.968925 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:54.968930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:54.968933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:54.968961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:54.970664 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.971169 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.971173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.971175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.971258 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:54.971689 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:54.971692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:54.971694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:54.971936 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:54.972008 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:54.972375 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 12:38:54.972509 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 12:38:54.973304 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:54.963422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34400)
0926 12:38:54.963900 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:54.963914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:54.964023 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:54.964502 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:54.964506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:54.964748 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:54.964905 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.964913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.965772 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.965776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.966706 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:54.966717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:54.967581 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:54.967585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:54.967685 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:54.968305 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:54.968321 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:54.968759 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:54.970796 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.970799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.970801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.971058 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.971395 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:54.971398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:54.971401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:54.971508 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:54.972080 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:54.972083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:54.972180 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 12:38:54.972933 (+ 753us) client_negotiation.cc:240] Negotiation successful
0926 12:38:54.973093 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":148,"threads_started":1}
I20260926 12:38:54.973891 17275 tablet_copy_service.cc:140] P c19fed68d8e04971bcf0fb5fd081c5e4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 747538dff7b84fb6864644aacd55ff47 ({username='oryx', principal='oryx/127.10.224.189@KRBTEST.COM'} at 127.0.0.1:34400)
I20260926 12:38:54.974129 17275 tablet_copy_service.cc:161] P c19fed68d8e04971bcf0fb5fd081c5e4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 747538dff7b84fb6864644aacd55ff47 at {username='oryx', principal='oryx/127.10.224.189@KRBTEST.COM'} at 127.0.0.1:34400: session id = 747538dff7b84fb6864644aacd55ff47-00000000000000000000000000000000
I20260926 12:38:54.976603 17275 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 12:38:54.978741 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 12:38:54.981523 17719 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: tablet copy: Starting download of 0 data blocks...
I20260926 12:38:54.981740 17719 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: tablet copy: Starting download of 1 WAL segments...
I20260926 12:38:54.983754 17719 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 12:38:54.986016 17275 tablet_copy_service.cc:342] P c19fed68d8e04971bcf0fb5fd081c5e4: Request end of tablet copy session 747538dff7b84fb6864644aacd55ff47-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.224.189@KRBTEST.COM'} at 127.0.0.1:34400
I20260926 12:38:54.986179 17275 tablet_copy_service.cc:433] P c19fed68d8e04971bcf0fb5fd081c5e4: ending tablet copy session 747538dff7b84fb6864644aacd55ff47-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 747538dff7b84fb6864644aacd55ff47
I20260926 12:38:54.994642 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:54.996688 17832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.996770 17829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:54.997385 17830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:54.997591 17719 server_base.cc:1034] running on GCE node
I20260926 12:38:54.997737 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:54.998064 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:54.999214 17719 hybrid_clock.cc:648] HybridClock initialized: now 1790426334999190 us; error 44 us; skew 500 ppm
Sep 26 12:38:54 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426334, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:55.000895 17719 init.cc:377] Logged in from keytab as oryx/127.10.224.189@KRBTEST.COM (short username oryx)
I20260926 12:38:55.001416 17719 webserver.cc:460] Webserver started at http://127.10.224.189:38585/ using document root <none> and password file <none>
I20260926 12:38:55.001624 17719 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:55.001667 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:55.002991 17719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:55.004014 17839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.004240 17719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 12:38:55.004360 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
uuid: "747538dff7b84fb6864644aacd55ff47"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "2538917fe387c945b59b69c6d04a3278"
server_key_iv: "6351810c6f88a5d66cee0b8b379d7116"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.004506 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:55.018618 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:55.021081 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:55.027014 17719 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.189:33525
I20260926 12:38:55.027222 17891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.189:33525 every 8 connection(s)
I20260926 12:38:55.027824 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
I20260926 12:38:55.029327 17892 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:38:55.030161 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Bootstrap starting.
I20260926 12:38:55.038319 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:38:55.038630 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Bootstrap complete.
I20260926 12:38:55.039141 17892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } }
I20260926 12:38:55.039269 17892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 747538dff7b84fb6864644aacd55ff47, State: Initialized, Role: LEARNER
I20260926 12:38:55.039386 17892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } }
I20260926 12:38:55.039764 17895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } }
I20260926 12:38:55.039929 17895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:38:55.039927 17892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:55.043102 17906 catalog_manager.cc:1259] Loaded cluster ID: ab7db6a15d1d43bcacdf3cea4b45ead3
I20260926 12:38:55.043207 17906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: loading cluster ID for follower catalog manager: success
I20260926 12:38:55.044909 17906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: acquiring CA information for follower catalog manager: success
I20260926 12:38:55.046123 17906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 12:38:55.119160 17828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:55.119479 17828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:55.119586 17828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:55.123970 17828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:55.124039 17828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:55.124060 17828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:55.124078 17828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:55.124094 17828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:55.129885 17828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:38019,127.10.224.189:33525,127.10.224.188:35243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.188.keytab
--principal=oryx/127.10.224.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.188:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.224.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:55.131094 17828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:55.132162 17446 heartbeater.cc:503] Master 127.10.224.190:38019 was elected leader, sending a full tablet report...
I20260926 12:38:55.132371 17828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:55.141878 17911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:55.141878 17914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:55.141878 17912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:55.142249 17828 server_base.cc:1034] running on GCE node
I20260926 12:38:55.142933 17828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:55.144322 17828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:55.145524 17828 hybrid_clock.cc:648] HybridClock initialized: now 1790426335145503 us; error 39 us; skew 500 ppm
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:55.148725 17828 init.cc:377] Logged in from keytab as oryx/127.10.224.188@KRBTEST.COM (short username oryx)
I20260926 12:38:55.149670 17828 webserver.cc:460] Webserver started at http://127.10.224.188:41627/ using document root <none> and password file <none>
I20260926 12:38:55.150228 17828 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:55.150290 17828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:55.150465 17828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:55.152678 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data/instance:
uuid: "fb639df2d8654bfabc75be8aeed5d93f"
format_stamp: "Formatted at 2026-09-26 12:38:55 on dist-test-slave-m3pj"
server_key: "c1de3f306098ad5070a3de127ef853e8"
server_key_iv: "4ccb6dfbeadcc8809d9be42c383e9b09"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.153239 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal/instance:
uuid: "fb639df2d8654bfabc75be8aeed5d93f"
format_stamp: "Formatted at 2026-09-26 12:38:55 on dist-test-slave-m3pj"
server_key: "c1de3f306098ad5070a3de127ef853e8"
server_key_iv: "4ccb6dfbeadcc8809d9be42c383e9b09"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.158056 17828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:55.160939 17921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.162456 17828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:55.162590 17828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
uuid: "fb639df2d8654bfabc75be8aeed5d93f"
format_stamp: "Formatted at 2026-09-26 12:38:55 on dist-test-slave-m3pj"
server_key: "c1de3f306098ad5070a3de127ef853e8"
server_key_iv: "4ccb6dfbeadcc8809d9be42c383e9b09"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.162694 17828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:55.185348 17828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:55.188953 17828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:55.189141 17828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:55.196414 17828 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.188:35243
I20260926 12:38:55.196724 17973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.188:35243 every 8 connection(s)
I20260926 12:38:55.197708 17828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
I20260926 12:38:55.200676 17974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:55.204902 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 17828
I20260926 12:38:55.204685 17974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.188@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.224.188@KRBTEST.COM
I20260926 12:38:55.220084 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.207137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.10.224.190:38019)
0926 12:38:55.207274 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.207279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.207589 (+ 310us) server_negotiation.cc:407] Connection header received
0926 12:38:55.208586 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.208590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.208654 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.208771 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.209730 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.210302 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.211969 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.212216 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.212339 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.215348 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.215351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.215353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.215383 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.217581 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.218335 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.218341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.218345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.218450 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.218849 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.218853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.218855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.219064 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.219169 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.219781 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.219917 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:38:55.221134 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.206944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34404)
0926 12:38:55.207428 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.207470 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.208261 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.208830 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.208836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.209246 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.209579 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.209591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.210447 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.210455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.211843 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.211853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.212400 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.212404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.212602 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.213311 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.213329 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.215136 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.217722 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.217730 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.217741 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.218212 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.218564 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.218571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.218573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.218725 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.219179 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.219188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.219474 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.219736 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.220098 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":60,"thread_start_us":184,"threads_started":1}
I20260926 12:38:55.221134 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.206809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.10.224.188:35243)
0926 12:38:55.207281 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.207303 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.207343 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:38:55.208229 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.208250 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.208467 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.208663 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.209567 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.210495 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.211840 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.212268 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.212459 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.216033 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.216044 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.216064 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.216096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.218315 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.218875 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.218882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.218884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.218977 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.219342 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.219345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.219346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.219654 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.219885 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.220100 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.220555 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260926 12:38:55.222707 17974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.188@KRBTEST.COM for oryx/127.10.224.189@KRBTEST.COM
I20260926 12:38:55.233024 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.223279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.0.0.1:46966)
0926 12:38:55.223465 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.223493 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.223626 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.224025 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.224027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.224188 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.224303 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.224308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.225352 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.225355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.226250 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.226261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.226720 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.226722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.226822 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.227456 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.227469 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.228910 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.231154 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.231158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.231159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.231446 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.231859 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.231862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.231864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.231956 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.232519 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.232522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.232632 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.232733 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.232875 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 12:38:55.233060 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.223391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.10.224.189:33525)
0926 12:38:55.223748 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.223758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.223771 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:38:55.223808 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.223810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.223862 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.223956 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.224534 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.225227 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.226377 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.226615 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.226703 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.229118 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.229121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.229124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.229165 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.231008 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.231646 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.231649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.231650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.231743 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.232093 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.232096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.232098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.232314 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.232410 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.232799 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.232928 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":152,"threads_started":1}
I20260926 12:38:55.234087 17974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.188@KRBTEST.COM for oryx/127.10.224.188@KRBTEST.COM
I20260926 12:38:55.242512 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.234631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.0.0.1:41678)
0926 12:38:55.234764 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.234781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.234885 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.235101 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.235104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.235289 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.235366 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.235371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.236195 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.236198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.236698 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.236705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.237158 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.237162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.237229 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.237759 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.237770 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.238988 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.241002 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.241005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.241006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.241272 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.241553 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.241555 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.241556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.241629 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.242035 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.242037 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.242119 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.242202 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.242321 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 12:38:55.242525 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.234800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.10.224.188:35243)
0926 12:38:55.234919 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.234925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.234941 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:38:55.234978 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.234980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.235011 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.235091 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.235476 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.236064 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.236808 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.237053 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.237159 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.239128 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.239131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.239133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.239152 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.240873 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.241375 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.241378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.241380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.241472 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.241704 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.241706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.241708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.241895 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.242035 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.242245 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.242383 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":56}
I20260926 12:38:55.246206 17974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Bootstrap starting.
I20260926 12:38:55.248939 17974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:55.249764 17974 log.cc:824] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:55.252640 17974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: No bootstrap required, opened a new log
I20260926 12:38:55.257863 17974 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } }
I20260926 12:38:55.258168 17974 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:55.258255 17974 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb639df2d8654bfabc75be8aeed5d93f, State: Initialized, Role: FOLLOWER
I20260926 12:38:55.258770 17974 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } }
I20260926 12:38:55.259312 17980 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } } }
I20260926 12:38:55.259444 17980 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:55.260285 17974 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: configured and running, proceeding with master startup.
W20260926 12:38:55.265528 17991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:55.265633 17991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:55.269160 17828 master_runner.cc:428] Detected that this master fb639df2d8654bfabc75be8aeed5d93f is joining an existing cluster
I20260926 12:38:55.269243 17828 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.224.188:35243
I20260926 12:38:55.274946 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 17317
I20260926 12:38:55.280390 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.271426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.10.224.188:35243)
0926 12:38:55.271626 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.271632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.272655 (+ 1023us) server_negotiation.cc:407] Connection header received
0926 12:38:55.272802 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.272805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.272855 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.272939 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.273489 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.274224 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.275458 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.275806 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.275868 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.277047 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.277051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.277053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.277075 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.278548 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.278967 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.278973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.278974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.279046 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.279301 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.279306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.279308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.279503 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.279648 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.280031 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.280178 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260926 12:38:55.280846 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.271563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.0.0.1:41694)
0926 12:38:55.272569 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.272584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.272706 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.272983 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.272986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.273236 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.273364 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.273372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.274333 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.274336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.275356 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.275363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.276007 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.276010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.276120 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.276557 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.276567 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.276916 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.278673 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.278677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.278679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.278877 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.279137 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.279139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.279141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.279219 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.279682 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.279685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.279767 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.280541 (+ 774us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.280672 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":3,"spinlock_wait_cycles":2048,"thread_start_us":98,"threads_started":1}
I20260926 12:38:55.286685 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:44163
--local_ip_for_outbound_sockets=127.10.224.129
--tserver_master_addrs=127.10.224.190:38019
--webserver_port=42339
--webserver_interface=127.10.224.129
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab
--principal=oryx/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525,127.10.224.188:35243 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:38:55.288543 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.272274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.10.224.189:33525)
0926 12:38:55.272408 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.272413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.280383 (+ 7970us) server_negotiation.cc:407] Connection header received
0926 12:38:55.281343 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.281347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.281402 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.281458 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.282016 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.282713 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.283683 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.283969 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.284032 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.285192 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.285196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.285199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.285230 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.286767 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.287270 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.287274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.287276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.287362 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.287697 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.287700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.287702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.287899 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.287960 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.288272 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.288400 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:38:55.288558 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.271950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.0.0.1:46968)
0926 12:38:55.280295 (+ 8345us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.280311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.280796 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.281598 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.281600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.281758 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.281850 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.281855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.282829 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.282833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.283504 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.283557 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.284157 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.284159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.284223 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.284719 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.284742 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.285053 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.286926 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.286930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.286934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.287174 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.287459 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.287462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.287464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.287582 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.288019 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.288021 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.288106 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.288204 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.288363 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7754,"thread_start_us":81,"threads_started":1}
I20260926 12:38:55.304543 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.271372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34416)
0926 12:38:55.271803 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.271816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.271964 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.272337 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.272340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.272620 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.272814 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.272820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.273686 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.273688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.274633 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.274642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.275367 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.275369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.275476 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.276101 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.276114 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.300090 (+ 23976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.302660 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.302666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.302669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.302975 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.303308 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.303311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.303313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.303409 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.303903 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.303906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.304031 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.304141 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.304315 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":27,"spinlock_wait_cycles":896,"thread_start_us":193,"threads_started":1}
I20260926 12:38:55.304545 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.271432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.10.224.190:38019)
0926 12:38:55.271610 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.271615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.271885 (+ 270us) server_negotiation.cc:407] Connection header received
0926 12:38:55.272074 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.272077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.272125 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.272177 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.272952 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.273552 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.274759 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.275111 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.275188 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.300401 (+ 25213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.300406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.300410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.300451 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.302498 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.303095 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.303099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.303101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.303205 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.303512 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.303516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.303541 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.303802 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.303924 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.304237 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.304373 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426333, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.190@KRBTEST.COM for oryx/127.10.224.188@KRBTEST.COM
I20260926 12:38:55.322886 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.309951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.0.0.1:41700)
0926 12:38:55.310164 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.310179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.310266 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.310589 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.310592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.310799 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.310927 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.310934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.312076 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.312080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.313085 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.313098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.313873 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.313877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.313996 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.314596 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.314613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.318047 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.320987 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.320991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.320993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.321324 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.321669 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.321672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.321674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.321773 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.322299 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.322302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.322423 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.322535 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.322710 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 12:38:55.322937 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.310044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.10.224.188:35243)
0926 12:38:55.310175 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.310179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.310224 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:38:55.310403 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.310408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.310469 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.310563 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.311054 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.311930 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.313239 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.313633 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.313734 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.318422 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.318428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.318431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.318463 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.320781 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.321440 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.321453 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.321456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.321561 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.321870 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.321874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.321876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.322182 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.322284 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.322608 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.322743 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:38:55.324579 17255 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.224.188:35243: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f"
member_type: NON_VOTER
last_known_addr {
host: "127.10.224.188"
port: 35243
}
attrs {
promote: true
}
}
dest_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4"
cas_config_opid_index: 5
I20260926 12:38:55.325271 17255 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } }
I20260926 12:38:55.326534 17816 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.328960 17816 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.224.188:35243
I20260926 12:38:55.329164 17816 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.329269 17816 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:55.330262 17948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:55.331892 17948 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Refusing update from remote peer c19fed68d8e04971bcf0fb5fd081c5e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 12:38:55.332724 17828 master.cc:537] Master@127.10.224.188:35243 shutting down...
I20260926 12:38:55.332710 18001 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:38:55.335911 17242 consensus_peers.cc:487] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 -> Peer fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188:35243): Couldn't send request to peer fb639df2d8654bfabc75be8aeed5d93f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 12:38:55.340641 17828 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:55.341018 17828 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:55.341141 17828 tablet_replica.cc:330] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: stopping tablet replica
I20260926 12:38:55.341661 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.327844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.0.0.1:46978)
0926 12:38:55.328026 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.328044 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.328141 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.328672 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.328675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.328693 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:55.328787 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.328793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.338158 (+ 9365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.338164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.339537 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.339554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.340940 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.340947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.341062 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.341313 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.341402 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.341509 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 12:38:55.342166 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.328164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.10.224.189:33525)
0926 12:38:55.328307 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.328311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.328326 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:55.328380 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.328383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.328443 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.328532 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:55.329185 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.337989 (+ 8804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.339724 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.340440 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.340579 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.340732 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.341942 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.342050 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:38:55.343133 17866 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 LEARNER]: Refusing update from remote peer c19fed68d8e04971bcf0fb5fd081c5e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 12:38:55.344177 18001 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 12:38:55.346393 17866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.346917 17828 master.cc:552] Master@127.10.224.188:35243 shutdown complete.
I20260926 12:38:55.346966 17828 master_runner.cc:305] Clearing existing system tablet
I20260926 12:38:55.347226 17828 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:55.347340 17828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:55.349380 17828 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:38:55.349375 17895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c19fed68d8e04971bcf0fb5fd081c5e4. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.349591 17895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:38:55.349390 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: NON_VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: true } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.349843 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:38:55.351056 18001 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: attempting to promote NON_VOTER 747538dff7b84fb6864644aacd55ff47 to VOTER
I20260926 12:38:55.351672 18001 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } }
I20260926 12:38:55.353571 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.352794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.0.0.1:41708)
0926 12:38:55.353385 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.353479 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484}
I20260926 12:38:55.354305 17866 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 LEARNER]: Refusing update from remote peer c19fed68d8e04971bcf0fb5fd081c5e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 12:38:55.355115 17816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 12:38:55.355767 18005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.356060 17828 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:55.356190 17828 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260926 12:38:55.356226 17828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
uuid: "fb639df2d8654bfabc75be8aeed5d93f"
format_stamp: "Formatted at 2026-09-26 12:38:55 on dist-test-slave-m3pj"
server_key: "c1de3f306098ad5070a3de127ef853e8"
server_key_iv: "4ccb6dfbeadcc8809d9be42c383e9b09"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.356784 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.356480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.0.0.1:41710)
0926 12:38:55.356637 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.356698 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260926 12:38:55.356942 17242 consensus_peers.cc:487] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 -> Peer fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188:35243): Couldn't send request to peer fb639df2d8654bfabc75be8aeed5d93f. Status: Network error: Client connection negotiation failed: client connection to 127.10.224.188:35243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 12:38:55.357987 17816 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 747538dff7b84fb6864644aacd55ff47 (127.10.224.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.358372 17828 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:55.360451 17866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 747538dff7b84fb6864644aacd55ff47 (127.10.224.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.361701 17828 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 12:38:55.361950 17828 log.cc:1196] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 12:38:55.362160 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.362403 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:55.362797 17828 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Deleting consensus metadata
I20260926 12:38:55.362762 17816 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.362892 17816 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:55.363160 17828 master_runner.cc:315] Copying system tablet from 127.10.224.190:38019
I20260926 12:38:55.363135 17816 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.363219 17816 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:55.364493 17828 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: tablet copy: Beginning tablet copy session from remote peer at address 127.10.224.190:38019
I20260926 12:38:55.374289 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.365060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34432)
0926 12:38:55.365324 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.365341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.365464 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.365768 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.365771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.366004 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.366137 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.366144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.367344 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.367348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.368278 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.368291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.368946 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.368950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.369097 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.369744 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.369759 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.370293 (+ 534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.372473 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.372477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.372480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.372755 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.373073 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.373076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.373078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.373176 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.373647 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.373651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.373763 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.373876 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.374069 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260926 12:38:55.374286 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.365404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.10.224.190:38019)
0926 12:38:55.365519 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.365525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.365540 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:55.365587 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.365591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.365639 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.365735 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.366267 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.367188 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.368415 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.368810 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.368940 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.370450 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.370454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.370458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.370489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.372338 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.372861 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.372865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.372867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.372972 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.373271 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.373275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.373277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.373542 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.373650 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.373961 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.374107 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:38:55.375129 17275 tablet_copy_service.cc:140] P c19fed68d8e04971bcf0fb5fd081c5e4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fb639df2d8654bfabc75be8aeed5d93f ({username='oryx', principal='oryx/127.10.224.188@KRBTEST.COM'} at 127.0.0.1:34432)
I20260926 12:38:55.375237 17275 tablet_copy_service.cc:161] P c19fed68d8e04971bcf0fb5fd081c5e4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fb639df2d8654bfabc75be8aeed5d93f at {username='oryx', principal='oryx/127.10.224.188@KRBTEST.COM'} at 127.0.0.1:34432: session id = fb639df2d8654bfabc75be8aeed5d93f-00000000000000000000000000000000
I20260926 12:38:55.376473 17275 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 12:38:55.377521 17580 heartbeater.cc:503] Master 127.10.224.190:38019 was elected leader, sending a full tablet report...
I20260926 12:38:55.377939 17828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 12:38:55.380632 17828 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: tablet copy: Starting download of 0 data blocks...
I20260926 12:38:55.380867 17828 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: tablet copy: Starting download of 1 WAL segments...
I20260926 12:38:55.382432 17828 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 12:38:55.384713 17275 tablet_copy_service.cc:342] P c19fed68d8e04971bcf0fb5fd081c5e4: Request end of tablet copy session fb639df2d8654bfabc75be8aeed5d93f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.224.188@KRBTEST.COM'} at 127.0.0.1:34432
I20260926 12:38:55.384821 17275 tablet_copy_service.cc:433] P c19fed68d8e04971bcf0fb5fd081c5e4: ending tablet copy session fb639df2d8654bfabc75be8aeed5d93f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fb639df2d8654bfabc75be8aeed5d93f
I20260926 12:38:55.394083 17828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:55.396145 18013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:55.396443 18016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:55.396490 17828 server_base.cc:1034] running on GCE node
W20260926 12:38:55.399703 18014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:55.400032 17828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:55.400406 17828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:55.401569 17828 hybrid_clock.cc:648] HybridClock initialized: now 1790426335401548 us; error 47 us; skew 500 ppm
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:55.403401 17828 init.cc:377] Logged in from keytab as oryx/127.10.224.188@KRBTEST.COM (short username oryx)
I20260926 12:38:55.403980 17828 webserver.cc:460] Webserver started at http://127.10.224.188:41223/ using document root <none> and password file <none>
I20260926 12:38:55.404186 17828 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:55.404232 17828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:55.405757 17828 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 12:38:55.406880 18023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.407246 17828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 12:38:55.407342 17828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
uuid: "fb639df2d8654bfabc75be8aeed5d93f"
format_stamp: "Formatted at 2026-09-26 12:38:55 on dist-test-slave-m3pj"
server_key: "c1de3f306098ad5070a3de127ef853e8"
server_key_iv: "4ccb6dfbeadcc8809d9be42c383e9b09"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.407442 17828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:55.420608 17828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:55.423739 17828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:55.429224 17828 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.188:35243
I20260926 12:38:55.429548 18075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.188:35243 every 8 connection(s)
I20260926 12:38:55.429931 17828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
I20260926 12:38:55.431984 18076 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:38:55.433243 18076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Bootstrap starting.
I20260926 12:38:55.446451 18076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:38:55.446907 18076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: Bootstrap complete.
I20260926 12:38:55.447676 18076 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } }
I20260926 12:38:55.447840 18076 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb639df2d8654bfabc75be8aeed5d93f, State: Initialized, Role: LEARNER
I20260926 12:38:55.447976 18076 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } }
I20260926 12:38:55.448386 18079 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.448526 18076 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:55.448695 18079 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: This master's current role is: LEARNER
I20260926 12:38:55.453230 18093 catalog_manager.cc:1259] Loaded cluster ID: ab7db6a15d1d43bcacdf3cea4b45ead3
I20260926 12:38:55.453298 18093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: loading cluster ID for follower catalog manager: success
I20260926 12:38:55.455840 18093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: acquiring CA information for follower catalog manager: success
I20260926 12:38:55.457196 18093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 12:38:55.457926 18000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:55.458262 18000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:55.458324 18000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:55.464799 18000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:55.464922 18000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:55.465021 18000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:38:55.473760 18000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab
--principal=oryx/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:44163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=42339
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525,127.10.224.188:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:55.475373 18000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:55.476953 18000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:55.489753 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:55.489790 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:55.489794 18095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:55.490000 18000 server_base.cc:1034] running on GCE node
I20260926 12:38:55.490515 18000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:55.491222 18000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:55.492432 18000 hybrid_clock.cc:648] HybridClock initialized: now 1790426335492408 us; error 29 us; skew 500 ppm
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:55.495657 18000 init.cc:377] Logged in from keytab as oryx/127.10.224.129@KRBTEST.COM (short username oryx)
I20260926 12:38:55.496709 18000 webserver.cc:460] Webserver started at http://127.10.224.129:42339/ using document root <none> and password file <none>
I20260926 12:38:55.497476 18000 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:55.497591 18000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:55.502316 18000 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:38:55.505034 18105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.506574 18000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:38:55.506696 18000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "4631bcece0204ad2a8252fc5fe1a0d94"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "aaee775c5df2b52b5b28a7633804cb9c"
server_key_iv: "fc69a72ad70985b12c80837dc8e7695f"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.507292 18000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:55.536466 18000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:55.540037 18000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:55.540215 18000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:55.540994 18000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:55.542026 18000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:55.542078 18000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.542124 18000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:55.542147 18000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.552510 18000 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:44163
I20260926 12:38:55.552537 18218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:44163 every 8 connection(s)
I20260926 12:38:55.553668 18000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:38:55.557363 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 18000
I20260926 12:38:55.557577 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 17451
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.189@KRBTEST.COM
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.188@KRBTEST.COM
I20260926 12:38:55.567297 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:41439
--local_ip_for_outbound_sockets=127.10.224.130
--tserver_master_addrs=127.10.224.190:38019
--webserver_port=39871
--webserver_interface=127.10.224.130
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab
--principal=oryx/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525,127.10.224.188:35243 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:38:55.573036 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.558207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:46165 (local address 127.10.224.189:33525)
0926 12:38:55.558332 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.558338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.558443 (+ 105us) server_negotiation.cc:407] Connection header received
0926 12:38:55.558902 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.558906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.558971 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.559059 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.560769 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.561526 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.562876 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.563145 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.563275 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.567893 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.567898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.567901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.567931 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.569975 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.570813 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.570817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.570821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.570919 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.571424 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.571428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.571431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.571731 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.571812 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.572687 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.572843 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:38:55.573057 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.555769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:54403 (local address 127.10.224.188:35243)
0926 12:38:55.556068 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.556074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.557356 (+ 1282us) server_negotiation.cc:407] Connection header received
0926 12:38:55.558939 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.558946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.559016 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.559085 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.560768 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.562064 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.563563 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.563988 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.564125 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.567731 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.567739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.567751 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.567793 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.570627 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.571199 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.571203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.571205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.571300 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.571663 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.571667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.571669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.571941 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.572058 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.572687 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.572866 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":113,"threads_started":1}
I20260926 12:38:55.575150 18226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.557830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.10.224.129:46165)
0926 12:38:55.558330 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.558347 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.558688 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.559271 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.559279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.559865 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.560597 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.560614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.561700 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.561703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.562730 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.562740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.563323 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.563327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.563598 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.564553 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.564584 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.567716 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.570153 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.570161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.570185 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.570672 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.571061 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.571065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.571067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.571281 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.571907 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.571916 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.572355 (+ 439us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.573303 (+ 948us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.573818 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":182,"spinlock_wait_cycles":9088,"thread_start_us":101,"threads_started":1}
I20260926 12:38:55.575428 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.556501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.10.224.129:54403)
0926 12:38:55.557159 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.557226 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.558687 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.559271 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.559282 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.559872 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.560599 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.560617 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.562258 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.562264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.563397 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.563410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.564127 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.564131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.564245 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.565295 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.565315 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.567437 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.570781 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.570787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.570790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.571074 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.571421 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.571424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.571426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.571544 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.572117 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.572120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.572355 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.573303 (+ 948us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.573997 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":294,"spinlock_wait_cycles":54528,"thread_start_us":163,"threads_started":1}
I20260926 12:38:55.576977 18221 heartbeater.cc:347] Connected to a master server at 127.10.224.189:33525
I20260926 12:38:55.577337 18221 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:55.578140 18221 heartbeater.cc:511] Master 127.10.224.189:33525 requested a full tablet report, sending...
I20260926 12:38:55.577018 18219 heartbeater.cc:347] Connected to a master server at 127.10.224.188:35243
I20260926 12:38:55.578945 18219 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:55.579180 18219 heartbeater.cc:511] Master 127.10.224.188:35243 requested a full tablet report, sending...
I20260926 12:38:55.580212 17856 ts_manager.cc:194] Registered new tserver with Master: 4631bcece0204ad2a8252fc5fe1a0d94 (127.10.224.129:44163)
I20260926 12:38:55.580302 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.555777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:55269 (local address 127.10.224.190:38019)
0926 12:38:55.555916 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.555921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.564576 (+ 8655us) server_negotiation.cc:407] Connection header received
0926 12:38:55.564775 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.564779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.564848 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.564955 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.565523 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.566395 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.567669 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.568013 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.568099 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.572141 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.572146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.572149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.572185 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.574749 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.575960 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.575963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.575967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.576091 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.576462 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.576468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.576472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.576755 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.576857 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.579898 (+ 3041us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.580076 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:38:55.580893 18040 ts_manager.cc:194] Registered new tserver with Master: 4631bcece0204ad2a8252fc5fe1a0d94 (127.10.224.129:44163)
I20260926 12:38:55.581182 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.556501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:55269)
0926 12:38:55.564473 (+ 7972us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.564492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.564659 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.564976 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.564982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.565232 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.565369 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.565380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.566549 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.566555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.567489 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.567502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.568297 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.568300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.568366 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.569114 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.569137 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.571920 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.574917 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.574922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.574927 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.575815 (+ 888us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.576224 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.576229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.576235 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.576342 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.576925 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.576930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.579583 (+ 2653us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.580468 (+ 885us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.580919 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":167,"threads_started":1}
I20260926 12:38:55.582248 18220 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38019
I20260926 12:38:55.582365 18220 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:55.582537 18220 heartbeater.cc:511] Master 127.10.224.190:38019 requested a full tablet report, sending...
I20260926 12:38:55.583344 17255 ts_manager.cc:194] Re-registered known tserver with Master: 4631bcece0204ad2a8252fc5fe1a0d94 (127.10.224.129:44163)
I20260926 12:38:55.583997 17255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:55269
I20260926 12:38:55.636224 17714 heartbeater.cc:503] Master 127.10.224.190:38019 was elected leader, sending a full tablet report...
W20260926 12:38:55.692932 18227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:55.693229 18227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:55.693274 18227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:55.697940 18227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:55.698017 18227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:55.698117 18227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:38:55.704030 18227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab
--principal=oryx/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:41439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=39871
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525,127.10.224.188:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:55.705336 18227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:55.706488 18227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:55.717944 18235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:55.717962 18233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:55.718041 18227 server_base.cc:1034] running on GCE node
W20260926 12:38:55.717962 18232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:55.718817 18227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:55.720067 18227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:55.726831 18227 hybrid_clock.cc:648] HybridClock initialized: now 1790426335726786 us; error 75 us; skew 500 ppm
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:55.730346 18227 init.cc:377] Logged in from keytab as oryx/127.10.224.130@KRBTEST.COM (short username oryx)
I20260926 12:38:55.731598 18227 webserver.cc:460] Webserver started at http://127.10.224.130:39871/ using document root <none> and password file <none>
I20260926 12:38:55.732280 18227 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:55.732405 18227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:55.737265 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.733141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.0.0.1:41720)
0926 12:38:55.733284 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.733314 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.733440 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.733787 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.733790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.733813 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:55.733934 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.733941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.734716 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.734720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.736052 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.736070 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.736891 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.736895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.736977 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.737001 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.737060 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.737141 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 12:38:55.737319 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.733204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.10.224.188:35243)
0926 12:38:55.733335 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.733339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.733385 (+ 46us) server_negotiation.cc:407] Connection header received
0926 12:38:55.733585 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.733589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.733656 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.733749 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:55.734046 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.734592 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.736214 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.736752 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.736891 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.737039 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.737085 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.737173 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:38:55.738122 18050 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 12:38:55.738309 18227 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:38:55.738567 18079 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c19fed68d8e04971bcf0fb5fd081c5e4. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:55.738708 18079 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: This master's current role is: LEARNER
I20260926 12:38:55.741299 18242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.742832 18227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:38:55.743007 18227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "ea61906bb2b54dfab8df4c3cae1b812a"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "6d17b76a6c0f288c57800d3820f697d8"
server_key_iv: "5b17cf5a08eb6a2eba0d329f75d35cb8"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.743664 18227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:55.762257 18227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:55.766012 18227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:55.766269 18227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:55.766966 18227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:55.767987 18227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:55.768059 18227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.768131 18227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:55.768177 18227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.778517 18227 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:41439
I20260926 12:38:55.778560 18355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:41439 every 8 connection(s)
I20260926 12:38:55.779920 18227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:38:55.786121 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 18227
I20260926 12:38:55.786335 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 17585
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.188@KRBTEST.COM
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.189@KRBTEST.COM
I20260926 12:38:55.793639 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:33013
--local_ip_for_outbound_sockets=127.10.224.131
--tserver_master_addrs=127.10.224.190:38019
--webserver_port=34175
--webserver_interface=127.10.224.131
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab
--principal=oryx/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525,127.10.224.188:35243 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:38:55.799113 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.782565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:34247 (local address 127.10.224.188:35243)
0926 12:38:55.782681 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.782686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.784300 (+ 1614us) server_negotiation.cc:407] Connection header received
0926 12:38:55.784827 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.784830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.784894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.784994 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.787685 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.788254 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.789232 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.789494 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.789591 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.793450 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.793455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.793458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.793491 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.795977 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.796770 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.796776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.796781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.796868 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.797715 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.797719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.797721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.797942 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.798087 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.798728 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.798879 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:38:55.799165 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.783952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:35705 (local address 127.10.224.189:33525)
0926 12:38:55.784058 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.784062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.784496 (+ 434us) server_negotiation.cc:407] Connection header received
0926 12:38:55.784707 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.784711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.784763 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.784819 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.786737 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.787658 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.789506 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.789822 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.789929 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.793869 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.793874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.793880 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.793914 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.796336 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.796980 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.796987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.796991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.797087 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.797412 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.797418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.797423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.797690 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.797767 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.798752 (+ 985us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.798930 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 12:38:55.801087 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.782700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.10.224.130:34247)
0926 12:38:55.784188 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.784203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.784483 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.785164 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.785172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.785681 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.787488 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.787504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.788376 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.788382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.789091 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.789099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.789768 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.789772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.789991 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.790669 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.790696 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.793235 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.796125 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.796133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.796157 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.796620 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.797010 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.797013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.797015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.797264 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.798181 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.798184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.798429 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.799311 (+ 882us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.799782 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"mutex_wait_us":1084,"spinlock_wait_cycles":35200,"thread_start_us":146,"threads_started":1}
I20260926 12:38:55.801682 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.783854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.10.224.130:35705)
0926 12:38:55.784414 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.784426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.784554 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.785164 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.785173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.785687 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.786567 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.786588 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.787853 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.787859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.789371 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.789385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.789957 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.789964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.790110 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.790820 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.790847 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.793677 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.796479 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.796484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.796486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.796778 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.797206 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.797212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.797217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.797310 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.797879 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.797890 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.798436 (+ 546us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.799860 (+ 1424us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.800047 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":77,"spinlock_wait_cycles":71296,"thread_start_us":72,"threads_started":1}
I20260926 12:38:55.802268 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.782700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:51777)
0926 12:38:55.783213 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:55.783268 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:55.784476 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:55.785164 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:55.785173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:55.785701 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:55.792155 (+ 6454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.792171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.793293 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.793297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.794195 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:55.794210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.795237 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.795242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:55.796208 (+ 966us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.796728 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:55.796744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:55.798392 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:55.800534 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.800536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.800538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.800766 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.801079 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:55.801084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:55.801086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:55.801197 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:55.801722 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:55.801724 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:55.801821 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 12:38:55.801939 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:38:55.802084 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":233,"spinlock_wait_cycles":12160,"thread_start_us":144,"threads_started":1}
I20260926 12:38:55.802271 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:55.782143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:51777 (local address 127.10.224.190:38019)
0926 12:38:55.782310 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:55.782316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:55.783393 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 12:38:55.784708 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:55.784712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:55.784763 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:55.784819 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:55.792313 (+ 7494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.793150 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.794494 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:55.795045 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:55.795193 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:55.798582 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:55.798586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:55.798590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:55.798624 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:55.800387 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.800880 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.800884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.800885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.800971 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:55.801338 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:55.801343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:55.801344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:55.801584 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:55.801667 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:55.802018 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 12:38:55.802135 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:38:55.802528 18356 heartbeater.cc:347] Connected to a master server at 127.10.224.188:35243
I20260926 12:38:55.802791 18356 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:55.802870 18357 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38019
I20260926 12:38:55.802933 18357 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:55.803467 18357 heartbeater.cc:511] Master 127.10.224.190:38019 requested a full tablet report, sending...
I20260926 12:38:55.803467 18356 heartbeater.cc:511] Master 127.10.224.188:35243 requested a full tablet report, sending...
I20260926 12:38:55.804129 18358 heartbeater.cc:347] Connected to a master server at 127.10.224.189:33525
I20260926 12:38:55.804210 18358 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:55.804352 18358 heartbeater.cc:511] Master 127.10.224.189:33525 requested a full tablet report, sending...
I20260926 12:38:55.805078 17255 ts_manager.cc:194] Re-registered known tserver with Master: ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439)
I20260926 12:38:55.805066 18040 ts_manager.cc:194] Registered new tserver with Master: ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439)
I20260926 12:38:55.805145 17856 ts_manager.cc:194] Registered new tserver with Master: ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439)
I20260926 12:38:55.805856 17255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:51777
W20260926 12:38:55.917053 18363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:55.917377 18363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:55.917481 18363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:55.921770 18363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:55.921859 18363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:55.921953 18363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:38:55.927721 18363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab
--principal=oryx/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:33013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=34175
--tserver_master_addrs=127.10.224.190:38019,127.10.224.189:33525,127.10.224.188:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:55.928925 18363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:55.930030 18363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:38:55.940646 18371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:55.940646 18369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:55.940694 18368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:55.940997 18363 server_base.cc:1034] running on GCE node
I20260926 12:38:55.941558 18363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:55.942339 18363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:55.943608 18363 hybrid_clock.cc:648] HybridClock initialized: now 1790426335943584 us; error 37 us; skew 500 ppm
Sep 26 12:38:55 dist-test-slave-m3pj krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:55.946794 18363 init.cc:377] Logged in from keytab as oryx/127.10.224.131@KRBTEST.COM (short username oryx)
I20260926 12:38:55.948032 18363 webserver.cc:460] Webserver started at http://127.10.224.131:34175/ using document root <none> and password file <none>
I20260926 12:38:55.948666 18363 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:55.948776 18363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:55.953173 18363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:38:55.955608 18378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:55.957095 18363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:55.957286 18363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "89039a18f4734d29ae5de038215d21aa"
format_stamp: "Formatted at 2026-09-26 12:38:54 on dist-test-slave-m3pj"
server_key: "72c184c8e7fb69adab335cf5b8e619f2"
server_key_iv: "697b300abf284ba6d8588a955e2d41cc"
server_key_version: "encryptionkey@0"
I20260926 12:38:55.957934 18363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:55.996779 18363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:56.000305 18363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:56.000520 18363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:56.001140 18363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:38:56.002045 18363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:38:56.002115 18363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:56.002187 18363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:38:56.002226 18363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:56.011910 18363 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:33013
I20260926 12:38:56.012002 18491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:33013 every 8 connection(s)
I20260926 12:38:56.013075 18363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:38:56.022871 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 18363
Sep 26 12:38:56 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.188@KRBTEST.COM
Sep 26 12:38:56 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
Sep 26 12:38:56 dist-test-slave-m3pj krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.189@KRBTEST.COM
I20260926 12:38:56.032191 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.016771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:39197 (local address 127.10.224.189:33525)
0926 12:38:56.016886 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.016890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.017032 (+ 142us) server_negotiation.cc:407] Connection header received
0926 12:38:56.017888 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.017892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.017944 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.018021 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.019455 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.020297 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.021863 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.022185 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.022306 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.027643 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.027647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.027650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.027684 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.029492 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.030034 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.030038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.030039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.030130 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.030618 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.030622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.030624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.030872 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.030967 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.031926 (+ 959us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.032041 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:38:56.033164 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.015404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.131:33697)
0926 12:38:56.016454 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:56.016467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:56.017342 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:56.017930 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:56.017935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:56.018514 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:56.019284 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.019300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.020211 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.020213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.021180 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.021189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.021733 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.021738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.021969 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.022735 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:56.022763 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:56.028314 (+ 5551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:56.030433 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:56.030436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:56.030438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:56.030676 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:56.031001 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:56.031004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:56.031006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:56.031089 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:56.031560 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:56.031563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:56.031654 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 12:38:56.031725 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:38:56.032042 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":140,"spinlock_wait_cycles":73856,"thread_start_us":176,"threads_started":1}
I20260926 12:38:56.033164 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.016618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.10.224.131:39197)
0926 12:38:56.016924 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:56.016947 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:56.017325 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:56.018059 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:56.018063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:56.018514 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:56.019286 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.019302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.020444 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.020448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.021704 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.021710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.022907 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.022911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.023027 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.023512 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:56.023552 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:56.027411 (+ 3859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:56.029624 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:56.029627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:56.029629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:56.029908 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:56.030244 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:56.030248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:56.030249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:56.030498 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:56.031017 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:56.031025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:56.031386 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 12:38:56.031643 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 12:38:56.032477 (+ 834us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":791,"spinlock_wait_cycles":11008,"thread_start_us":110,"threads_started":1}
I20260926 12:38:56.033838 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.014972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:33697 (local address 127.10.224.190:38019)
0926 12:38:56.015128 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.015133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.016573 (+ 1440us) server_negotiation.cc:407] Connection header received
0926 12:38:56.017582 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.017586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.017650 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.017751 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.019472 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.020058 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.021317 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.021550 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.021663 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.028528 (+ 6865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.028536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.028541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.028583 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.030320 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.030791 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.030795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.030796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.030881 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.031206 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.031209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.031211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.031412 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.033447 (+ 2035us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.033537 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.033671 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:38:56.034653 18494 heartbeater.cc:347] Connected to a master server at 127.10.224.189:33525
I20260926 12:38:56.034891 18494 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:56.035110 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.015844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:42915 (local address 127.10.224.188:35243)
0926 12:38:56.015982 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.015987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.016307 (+ 320us) server_negotiation.cc:407] Connection header received
0926 12:38:56.017581 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.017585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.017650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.017728 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.019465 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.020297 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.021835 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.022080 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.022141 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.025598 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.025604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.025608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.025649 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.027497 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.028600 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.028604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.028606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.028692 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.033780 (+ 5088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.033784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.033786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.034040 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.034145 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.034780 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.034929 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:38:56.035548 18494 heartbeater.cc:511] Master 127.10.224.189:33525 requested a full tablet report, sending...
I20260926 12:38:56.036906 17856 ts_manager.cc:194] Registered new tserver with Master: 89039a18f4734d29ae5de038215d21aa (127.10.224.131:33013)
I20260926 12:38:56.037302 18493 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38019
I20260926 12:38:56.037395 18493 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:56.037572 18493 heartbeater.cc:511] Master 127.10.224.190:38019 requested a full tablet report, sending...
I20260926 12:38:56.037889 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.015404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.10.224.131:42915)
0926 12:38:56.016088 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:56.016163 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:56.017325 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:56.017927 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:56.017934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:56.018517 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:56.019285 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.019302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.020444 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.020448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.021673 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.021696 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.022302 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.022308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.022429 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.022898 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:56.022913 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:56.025337 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:56.027672 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:56.027679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:56.027695 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:56.028052 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:56.033544 (+ 5492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:56.033548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:56.033550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:56.033649 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:56.034192 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:56.034195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:56.034298 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 12:38:56.034394 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 12:38:56.034560 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":290,"spinlock_wait_cycles":5248,"thread_start_us":725,"threads_started":1}
I20260926 12:38:56.038564 17255 ts_manager.cc:194] Re-registered known tserver with Master: 89039a18f4734d29ae5de038215d21aa (127.10.224.131:33013)
I20260926 12:38:56.038687 18492 heartbeater.cc:347] Connected to a master server at 127.10.224.188:35243
I20260926 12:38:56.038769 18492 heartbeater.cc:464] Registering TS with master...
I20260926 12:38:56.038925 18492 heartbeater.cc:511] Master 127.10.224.188:35243 requested a full tablet report, sending...
I20260926 12:38:56.039268 17255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:33697
I20260926 12:38:56.039692 18040 ts_manager.cc:194] Registered new tserver with Master: 89039a18f4734d29ae5de038215d21aa (127.10.224.131:33013)
I20260926 12:38:56.045563 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.029330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.10.224.188:35243)
0926 12:38:56.035288 (+ 5958us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.035293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.035308 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:56.035345 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.035349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.035395 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.035503 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.036249 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.037003 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.038127 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.038479 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.038588 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.040945 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.040949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.040952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.040980 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.043228 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.043803 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.043807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.043809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.043898 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.044211 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.044217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.044220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.044440 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.044580 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.045238 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.045359 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5891,"thread_start_us":72,"threads_started":1}
I20260926 12:38:56.046018 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.031432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.10.224.189:33525)
0926 12:38:56.032080 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.032084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.034818 (+ 2734us) server_negotiation.cc:407] Connection header received
0926 12:38:56.034860 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.034862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.034916 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.035012 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.036754 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.037521 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.038642 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.038952 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.039068 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.041893 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.041897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.041900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.041929 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.044059 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.044517 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.044520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.044522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.044632 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.044906 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.044909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.044911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.045160 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.045275 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.045738 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.045860 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":102,"threads_started":1}
I20260926 12:38:56.048147 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.030697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.10.224.190:38019)
0926 12:38:56.031953 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.031957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.032738 (+ 781us) server_negotiation.cc:407] Connection header received
0926 12:38:56.032781 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.032784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.032833 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.032911 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.034672 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.035928 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.037146 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.042421 (+ 5275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.042529 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.044181 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.044185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.044188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.044214 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.046347 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.046816 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.046821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.046826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.046922 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.047247 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.047250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.047251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.047494 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.047661 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.047878 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.047987 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":79,"threads_started":1}
I20260926 12:38:56.051149 17255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:56.054481 17255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:56.057257 18511 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: NON_VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: true } } }
I20260926 12:38:56.057439 18511 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:56.060710 18511 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: attempting to promote NON_VOTER fb639df2d8654bfabc75be8aeed5d93f to VOTER
I20260926 12:38:56.061091 18511 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } }
I20260926 12:38:56.062600 17866 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Refusing update from remote peer c19fed68d8e04971bcf0fb5fd081c5e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 12:38:56.063020 18512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 12:38:56.063400 18050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 LEARNER]: Refusing update from remote peer c19fed68d8e04971bcf0fb5fd081c5e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 12:38:56.063992 18512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 12:38:56.064975 18510 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:56.065493 17866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:56.066555 18510 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:56.066653 18510 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:56.066799 18510 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:56.066864 18510 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:56.066490 18050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:56.066980 18510 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:56.067052 18510 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:56.067857 18520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:56.068066 18520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:56.069429 18079 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:56.069550 18079 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:56.076411 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.071334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44163 (local address 127.0.0.1:58544)
0926 12:38:56.072010 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:56.072023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:56.072109 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:56.072559 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:56.072562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:56.072585 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:56.072691 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.072702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.073875 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.073879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.074888 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.074897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.076001 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.076005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.076085 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.076107 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:38:56.076170 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 12:38:56.076250 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":30,"thread_start_us":100,"threads_started":1}
I20260926 12:38:56.076491 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.071788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.10.224.129:44163)
0926 12:38:56.072120 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.072136 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.072161 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:38:56.072236 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.072242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.072427 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.072599 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:56.072794 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.073770 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.075028 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.075894 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.076026 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.076191 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.076243 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.076338 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":110,"threads_started":1}
I20260926 12:38:56.077504 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.071006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:33013 (local address 127.0.0.1:43420)
0926 12:38:56.071626 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:56.071641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:56.071736 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:56.072886 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:56.072889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:56.072907 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:56.072998 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.073004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.074561 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.074565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.075796 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.075806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.076978 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.076982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.077070 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.077094 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:38:56.077143 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 12:38:56.077210 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":26,"thread_start_us":103,"threads_started":1}
I20260926 12:38:56.077805 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.070316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41439 (local address 127.0.0.1:35692)
0926 12:38:56.070442 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:56.070459 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:56.070543 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:56.072203 (+ 1660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:56.072209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:56.072231 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:56.072344 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.072365 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.074580 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.074583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.075438 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.075446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.077403 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.077407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.077502 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.077529 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:38:56.077616 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:38:56.077692 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":101}
I20260926 12:38:56.077982 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.070500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.10.224.130:41439)
0926 12:38:56.071201 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.071215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.071242 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:38:56.071330 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.071336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.071642 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.072522 (+ 880us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:56.072963 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.074440 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.075600 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.077280 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.077416 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.077544 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.077734 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.077838 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":128,"threads_started":1}
I20260926 12:38:56.078138 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.071959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.10.224.131:33013)
0926 12:38:56.072353 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.072365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.072389 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:38:56.072490 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.072495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.072712 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.073163 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:56.073314 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.074431 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.075927 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.076715 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.077724 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.077845 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.077904 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.077996 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":104,"threads_started":1}
I20260926 12:38:56.079985 18153 tablet_service.cc:1467] Processing CreateTablet for tablet 04a0ae7e87ef4d358785b6959217967a (DEFAULT_TABLE table=test-table [id=764433442e58434399441d512015005e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:56.080464 18426 tablet_service.cc:1467] Processing CreateTablet for tablet 04a0ae7e87ef4d358785b6959217967a (DEFAULT_TABLE table=test-table [id=764433442e58434399441d512015005e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:56.081229 18153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a0ae7e87ef4d358785b6959217967a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:56.081437 18426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a0ae7e87ef4d358785b6959217967a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:56.081552 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 04a0ae7e87ef4d358785b6959217967a (DEFAULT_TABLE table=test-table [id=764433442e58434399441d512015005e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:56.082613 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a0ae7e87ef4d358785b6959217967a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:56.088382 18526 tablet_bootstrap.cc:492] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Bootstrap starting.
I20260926 12:38:56.089406 18527 tablet_bootstrap.cc:492] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Bootstrap starting.
I20260926 12:38:56.091250 18526 tablet_bootstrap.cc:654] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:56.091635 18527 tablet_bootstrap.cc:654] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:56.091805 18528 tablet_bootstrap.cc:492] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Bootstrap starting.
I20260926 12:38:56.092594 18527 log.cc:824] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:56.092664 18526 log.cc:824] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:56.093839 18528 tablet_bootstrap.cc:654] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:56.094539 18528 log.cc:824] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:56.094744 18527 tablet_bootstrap.cc:492] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: No bootstrap required, opened a new log
I20260926 12:38:56.094745 18526 tablet_bootstrap.cc:492] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: No bootstrap required, opened a new log
I20260926 12:38:56.094949 18527 ts_tablet_manager.cc:1397] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 12:38:56.094949 18526 ts_tablet_manager.cc:1397] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 12:38:56.096477 18528 tablet_bootstrap.cc:492] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: No bootstrap required, opened a new log
I20260926 12:38:56.096745 18528 ts_tablet_manager.cc:1397] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:38:56.099184 18527 raft_consensus.cc:348] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.099550 18527 raft_consensus.cc:374] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:56.099622 18527 raft_consensus.cc:729] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4631bcece0204ad2a8252fc5fe1a0d94, State: Initialized, Role: FOLLOWER
I20260926 12:38:56.100224 18527 consensus_queue.cc:260] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.100312 18526 raft_consensus.cc:348] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.100629 18526 raft_consensus.cc:374] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:56.100708 18526 raft_consensus.cc:729] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89039a18f4734d29ae5de038215d21aa, State: Initialized, Role: FOLLOWER
I20260926 12:38:56.101545 18527 ts_tablet_manager.cc:1428] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 12:38:56.101397 18528 raft_consensus.cc:348] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.101701 18528 raft_consensus.cc:374] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:56.101758 18528 raft_consensus.cc:729] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea61906bb2b54dfab8df4c3cae1b812a, State: Initialized, Role: FOLLOWER
I20260926 12:38:56.102313 18220 heartbeater.cc:503] Master 127.10.224.190:38019 was elected leader, sending a full tablet report...
I20260926 12:38:56.102299 18528 consensus_queue.cc:260] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.104437 18528 ts_tablet_manager.cc:1428] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 12:38:56.104862 18357 heartbeater.cc:503] Master 127.10.224.190:38019 was elected leader, sending a full tablet report...
I20260926 12:38:56.104970 18526 consensus_queue.cc:260] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.106734 18493 heartbeater.cc:503] Master 127.10.224.190:38019 was elected leader, sending a full tablet report...
I20260926 12:38:56.108394 18526 ts_tablet_manager.cc:1428] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
W20260926 12:38:56.264972 18495 tablet.cc:2369] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:56.273665 18534 raft_consensus.cc:482] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:56.273940 18534 raft_consensus.cc:504] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.275177 18534 leader_election.cc:288] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439), 4631bcece0204ad2a8252fc5fe1a0d94 (127.10.224.129:44163)
I20260926 12:38:56.279117 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.275638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:58661 (local address 127.10.224.129:44163)
0926 12:38:56.275759 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.275762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.275778 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:38:56.275955 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.275961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.276030 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.276092 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:56.276418 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.276990 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.278123 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.278520 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.278618 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.278698 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.278895 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.278981 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:38:56.279117 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.275453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41439 (local address 127.10.224.131:60257)
0926 12:38:56.275674 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:56.275689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:56.275813 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:56.276193 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:56.276197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:56.276213 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:56.276314 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.276319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.277175 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.277177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.277995 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.278004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.278700 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.278706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.278805 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.278822 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:38:56.278916 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:38:56.278979 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":25,"spinlock_wait_cycles":9984}
I20260926 12:38:56.279155 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.275524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:60257 (local address 127.10.224.130:41439)
0926 12:38:56.275690 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.275695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.275747 (+ 52us) server_negotiation.cc:407] Connection header received
0926 12:38:56.276016 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.276018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.276060 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.276162 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:56.276499 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.277020 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.278142 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.278556 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.278691 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.278745 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.278952 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.279044 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:38:56.279125 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.275487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44163 (local address 127.10.224.131:58661)
0926 12:38:56.275691 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:56.275706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:56.275815 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:56.276135 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:56.276140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:56.276163 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:56.276304 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.276309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.277125 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.277129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.277994 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:56.278004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.278647 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.278652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:56.278745 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.278777 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 12:38:56.278849 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 12:38:56.278947 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":10,"spinlock_wait_cycles":13952}
I20260926 12:38:56.279925 18173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0ae7e87ef4d358785b6959217967a" candidate_uuid: "89039a18f4734d29ae5de038215d21aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" is_pre_election: true
I20260926 12:38:56.279919 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0ae7e87ef4d358785b6959217967a" candidate_uuid: "89039a18f4734d29ae5de038215d21aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" is_pre_election: true
I20260926 12:38:56.280264 18310 raft_consensus.cc:2454] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89039a18f4734d29ae5de038215d21aa in term 0.
I20260926 12:38:56.280264 18173 raft_consensus.cc:2454] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89039a18f4734d29ae5de038215d21aa in term 0.
I20260926 12:38:56.280781 18382 leader_election.cc:302] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4631bcece0204ad2a8252fc5fe1a0d94, 89039a18f4734d29ae5de038215d21aa, ea61906bb2b54dfab8df4c3cae1b812a; no voters:
I20260926 12:38:56.281090 18534 raft_consensus.cc:2781] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:56.281188 18534 raft_consensus.cc:482] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:56.281262 18534 raft_consensus.cc:3037] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Advancing to term 1
W20260926 12:38:56.282253 18359 tablet.cc:2369] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:56.282502 18534 raft_consensus.cc:504] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.282853 18534 leader_election.cc:288] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [CANDIDATE]: Term 1 election: Requested vote from peers ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439), 4631bcece0204ad2a8252fc5fe1a0d94 (127.10.224.129:44163)
I20260926 12:38:56.283258 18173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0ae7e87ef4d358785b6959217967a" candidate_uuid: "89039a18f4734d29ae5de038215d21aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4631bcece0204ad2a8252fc5fe1a0d94"
I20260926 12:38:56.283368 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a0ae7e87ef4d358785b6959217967a" candidate_uuid: "89039a18f4734d29ae5de038215d21aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea61906bb2b54dfab8df4c3cae1b812a"
I20260926 12:38:56.283380 18173 raft_consensus.cc:3037] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:56.283499 18310 raft_consensus.cc:3037] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:56.284519 18173 raft_consensus.cc:2454] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89039a18f4734d29ae5de038215d21aa in term 1.
I20260926 12:38:56.284523 18310 raft_consensus.cc:2454] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89039a18f4734d29ae5de038215d21aa in term 1.
I20260926 12:38:56.284829 18379 leader_election.cc:302] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4631bcece0204ad2a8252fc5fe1a0d94, 89039a18f4734d29ae5de038215d21aa, ea61906bb2b54dfab8df4c3cae1b812a; no voters:
I20260926 12:38:56.284999 18534 raft_consensus.cc:2781] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:56.285216 18534 raft_consensus.cc:686] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 1 LEADER]: Becoming Leader. State: Replica: 89039a18f4734d29ae5de038215d21aa, State: Running, Role: LEADER
I20260926 12:38:56.285590 18534 consensus_queue.cc:237] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } }
I20260926 12:38:56.289808 17255 catalog_manager.cc:5495] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 89039a18f4734d29ae5de038215d21aa (127.10.224.131). New cstate: current_term: 1 leader_uuid: "89039a18f4734d29ae5de038215d21aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } health_report { overall_health: UNKNOWN } } }
W20260926 12:38:56.306712 18222 tablet.cc:2369] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:38:56.313457 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.309217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.10.224.131:33013)
0926 12:38:56.309416 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.309421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.309435 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:38:56.309516 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.309519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.309577 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.309675 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:56.309921 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.310487 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.311522 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.311787 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.311876 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.312322 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:56.312962 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:56.313091 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.313160 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.313223 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:38:56.320611 18310 raft_consensus.cc:1261] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 1 FOLLOWER]: Refusing update from remote peer 89039a18f4734d29ae5de038215d21aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:56.320611 18173 raft_consensus.cc:1261] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 1 FOLLOWER]: Refusing update from remote peer 89039a18f4734d29ae5de038215d21aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:56.321593 18535 consensus_queue.cc:1035] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:56.321810 18534 consensus_queue.cc:1035] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:56.330300 18538 mvcc.cc:205] Tried to move back new op lower bound from 7333586273559851008 to 7333586273428393984. Current Snapshot: MvccSnapshot[applied={T|T < 7333586273559851008}]
I20260926 12:38:56.331084 18540 mvcc.cc:205] Tried to move back new op lower bound from 7333586273559851008 to 7333586273428393984. Current Snapshot: MvccSnapshot[applied={T|T < 7333586273559851008}]
I20260926 12:38:56.345515 17253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:56.345723 17253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:56.350951 17253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c19fed68d8e04971bcf0fb5fd081c5e4: Sending DeleteTablet for 3 replicas of tablet 04a0ae7e87ef4d358785b6959217967a
I20260926 12:38:56.351831 18426 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a0ae7e87ef4d358785b6959217967a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:56 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:43420
I20260926 12:38:56.352185 18153 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a0ae7e87ef4d358785b6959217967a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:56 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:58544
I20260926 12:38:56.352653 18290 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a0ae7e87ef4d358785b6959217967a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:56 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:35692
I20260926 12:38:56.353572 18550 tablet_replica.cc:330] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: stopping tablet replica
I20260926 12:38:56.353879 18549 tablet_replica.cc:330] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: stopping tablet replica
I20260926 12:38:56.354194 18549 raft_consensus.cc:2229] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:56.354398 18550 raft_consensus.cc:2229] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:56.354823 18549 raft_consensus.cc:2258] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:56.354836 18550 raft_consensus.cc:2258] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:56.355880 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 17223
I20260926 12:38:56.356829 18550 ts_tablet_manager.cc:1905] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:56.359134 18550 ts_tablet_manager.cc:1918] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:56.359261 18550 log.cc:1196] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/04a0ae7e87ef4d358785b6959217967a
I20260926 12:38:56.359642 18550 ts_tablet_manager.cc:1939] T 04a0ae7e87ef4d358785b6959217967a P ea61906bb2b54dfab8df4c3cae1b812a: Deleting consensus metadata
I20260926 12:38:56.360471 18548 tablet_replica.cc:330] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: stopping tablet replica
I20260926 12:38:56.360939 18548 raft_consensus.cc:2229] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:38:56.361433 18548 raft_consensus.cc:2258] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:56.363379 18548 ts_tablet_manager.cc:1905] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:56.365583 18549 ts_tablet_manager.cc:1905] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:56.367143 18548 ts_tablet_manager.cc:1918] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:56.367354 18548 log.cc:1196] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/04a0ae7e87ef4d358785b6959217967a
I20260926 12:38:56.367831 18548 ts_tablet_manager.cc:1939] T 04a0ae7e87ef4d358785b6959217967a P 89039a18f4734d29ae5de038215d21aa: Deleting consensus metadata
I20260926 12:38:56.368412 18549 ts_tablet_manager.cc:1918] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:56.368521 18549 log.cc:1196] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/04a0ae7e87ef4d358785b6959217967a
I20260926 12:38:56.368860 18549 ts_tablet_manager.cc:1939] T 04a0ae7e87ef4d358785b6959217967a P 4631bcece0204ad2a8252fc5fe1a0d94: Deleting consensus metadata
W20260926 12:38:56.369983 18107 connection.cc:441] server connection from 127.0.0.1:58544 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58544 (request call id 1) could send its response
W20260926 12:38:56.370464 18381 connection.cc:535] server connection from 127.0.0.1:43420 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 12:38:56.370556 18246 connection.cc:535] server connection from 127.0.0.1:35692 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:38:56.372267 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.354167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.10.224.189:33525)
0926 12:38:56.354288 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.354293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.360096 (+ 5803us) server_negotiation.cc:407] Connection header received
0926 12:38:56.360290 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.360293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.360360 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.360465 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.362918 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.363721 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.364968 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.365357 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.365456 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.367802 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.367807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.367810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.367848 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.369723 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.370494 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.370498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.370501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.370613 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.370969 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.370972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.370974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.371263 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.371397 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.371951 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.372130 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:38:56.383821 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.354500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.10.224.188:35243)
0926 12:38:56.354606 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.354610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.372977 (+ 18367us) server_negotiation.cc:407] Connection header received
0926 12:38:56.373183 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.373186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.373244 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.373305 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.374261 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.375088 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.376071 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.376398 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.376517 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.379072 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.379076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.379078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.379103 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.381083 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.382235 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.382240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.382253 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.382355 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.382717 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.382723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.382725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.382963 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.383077 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.383461 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.383625 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:38:56.386982 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.376330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.10.224.189:33525)
0926 12:38:56.376479 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.376484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.376917 (+ 433us) server_negotiation.cc:407] Connection header received
0926 12:38:56.377117 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.377122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.377182 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.377242 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.378241 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.378947 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.380729 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.381107 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.381188 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.383166 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.383170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.383173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.383201 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.385283 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.385707 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.385711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.385713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.385802 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.386083 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.386087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.386088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.386279 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.386422 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.386667 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.386781 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:38:56.388336 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:56.376753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.10.224.188:35243)
0926 12:38:56.376870 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:56.376874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:56.377340 (+ 466us) server_negotiation.cc:407] Connection header received
0926 12:38:56.377502 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:56.377505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:56.377549 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:56.377604 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:56.378456 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.379231 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.380556 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:56.380917 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:56.380989 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:56.383019 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:56.383024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:56.383026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:56.383051 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:56.386254 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.386774 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.386778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.386780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.386863 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:56.387151 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:56.387155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:56.387157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:56.387385 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:56.387489 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:56.388022 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 12:38:56.388154 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:38:57.334673 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.334245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:54627)
0926 12:38:57.334498 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.334579 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
W20260926 12:38:57.334975 18220 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:38019 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.343147 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.342679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:45659)
0926 12:38:57.342960 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.343048 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
W20260926 12:38:57.343463 18357 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:38019 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.343777 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.343326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.131:57955)
0926 12:38:57.343572 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.343687 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
W20260926 12:38:57.344029 18493 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:38019 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.852645 18562 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c19fed68d8e04971bcf0fb5fd081c5e4)
I20260926 12:38:57.852849 18562 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } }
I20260926 12:38:57.853149 18563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c19fed68d8e04971bcf0fb5fd081c5e4)
I20260926 12:38:57.853333 18563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } }
I20260926 12:38:57.854058 18562 leader_election.cc:288] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019), 747538dff7b84fb6864644aacd55ff47 (127.10.224.189:33525)
I20260926 12:38:57.854566 18563 leader_election.cc:288] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019), fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188:35243)
I20260926 12:38:57.854803 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.854313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34916)
0926 12:38:57.854596 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.854680 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":65,"threads_started":1}
W20260926 12:38:57.854975 17840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.855118 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.854218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34902)
0926 12:38:57.854809 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.854968 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":94,"threads_started":1}
W20260926 12:38:57.855371 18024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.855947 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.855690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34918)
0926 12:38:57.855827 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.855872 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260926 12:38:57.856086 17840 leader_election.cc:334] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019): Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.856415 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.856091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34930)
0926 12:38:57.856307 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.856356 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260926 12:38:57.856580 18024 leader_election.cc:334] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019): Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.859205 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.854781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.10.224.189:33525)
0926 12:38:57.855003 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:57.855008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:57.855036 (+ 28us) server_negotiation.cc:407] Connection header received
0926 12:38:57.855086 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:57.855089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:57.855141 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:57.855237 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:57.855681 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:57.856395 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:57.857964 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:57.858559 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:57.858680 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:57.858732 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:57.858929 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 12:38:57.859043 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":62,"threads_started":1}
I20260926 12:38:57.859234 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.854546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:33525 (local address 127.0.0.1:47008)
0926 12:38:57.854809 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.854823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:57.854929 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:57.855385 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:57.855388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:57.855410 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:57.855544 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:57.855554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:57.856507 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:57.856513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:57.857824 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:57.857834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:57.858678 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:57.858681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:57.858778 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:57.858806 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:38:57.858917 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 12:38:57.859055 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":43,"threads_started":1}
I20260926 12:38:57.859246 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.854531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.10.224.188:35243)
0926 12:38:57.855406 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:57.855410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:57.855423 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:38:57.855466 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:57.855468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:57.855508 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:57.855610 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:57.855978 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:57.856696 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:57.858087 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:57.858700 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:57.858805 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:57.858836 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:57.859031 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 12:38:57.859099 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":53,"threads_started":1}
I20260926 12:38:57.859232 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.854346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:35243 (local address 127.0.0.1:41758)
0926 12:38:57.855129 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.855144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:57.855223 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:57.855720 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:57.855725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:57.855744 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:57.855842 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:57.855847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:57.856799 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:57.856803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:57.857983 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:57.857991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:57.858807 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:57.858811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:57.858915 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:57.858937 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:38:57.859007 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:38:57.859080 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":71,"threads_started":1}
I20260926 12:38:57.860039 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb639df2d8654bfabc75be8aeed5d93f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "747538dff7b84fb6864644aacd55ff47" is_pre_election: true
I20260926 12:38:57.860044 18050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "747538dff7b84fb6864644aacd55ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb639df2d8654bfabc75be8aeed5d93f" is_pre_election: true
I20260926 12:38:57.860266 17866 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb639df2d8654bfabc75be8aeed5d93f in term 1.
I20260926 12:38:57.860266 18050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747538dff7b84fb6864644aacd55ff47 in term 1.
I20260926 12:38:57.860670 18024 leader_election.cc:302] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 747538dff7b84fb6864644aacd55ff47, fb639df2d8654bfabc75be8aeed5d93f; no voters: c19fed68d8e04971bcf0fb5fd081c5e4
I20260926 12:38:57.860888 18562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 12:38:57.860975 18562 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Starting leader election (detected failure of leader c19fed68d8e04971bcf0fb5fd081c5e4)
I20260926 12:38:57.861032 18562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:38:57.861203 17843 leader_election.cc:302] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 747538dff7b84fb6864644aacd55ff47, fb639df2d8654bfabc75be8aeed5d93f; no voters: c19fed68d8e04971bcf0fb5fd081c5e4
I20260926 12:38:57.861413 18563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 12:38:57.861469 18563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c19fed68d8e04971bcf0fb5fd081c5e4)
I20260926 12:38:57.861515 18563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:38:57.862316 18562 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } }
I20260926 12:38:57.862625 18563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } }
I20260926 12:38:57.862701 18562 leader_election.cc:288] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [CANDIDATE]: Term 2 election: Requested vote from peers c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019), 747538dff7b84fb6864644aacd55ff47 (127.10.224.189:33525)
I20260926 12:38:57.863013 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.862787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34948)
0926 12:38:57.862872 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.862928 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 12:38:57.863039 18563 leader_election.cc:288] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 2 election: Requested vote from peers c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019), fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188:35243)
I20260926 12:38:57.863126 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb639df2d8654bfabc75be8aeed5d93f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "747538dff7b84fb6864644aacd55ff47"
I20260926 12:38:57.863261 17866 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fb639df2d8654bfabc75be8aeed5d93f in current term 2: Already voted for candidate 747538dff7b84fb6864644aacd55ff47 in this term.
I20260926 12:38:57.863791 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.863569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34950)
0926 12:38:57.863660 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.863721 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 12:38:57.863945 18050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "747538dff7b84fb6864644aacd55ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb639df2d8654bfabc75be8aeed5d93f"
I20260926 12:38:57.864041 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.863783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34952)
0926 12:38:57.863929 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.863980 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 12:38:57.864096 18050 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 747538dff7b84fb6864644aacd55ff47 in current term 2: Already voted for candidate fb639df2d8654bfabc75be8aeed5d93f in this term.
W20260926 12:38:57.864125 18024 leader_election.cc:334] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019): Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.864180 18024 leader_election.cc:302] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb639df2d8654bfabc75be8aeed5d93f; no voters: 747538dff7b84fb6864644aacd55ff47, c19fed68d8e04971bcf0fb5fd081c5e4
I20260926 12:38:57.864355 18562 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 12:38:57.864732 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:57.864322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:34956)
0926 12:38:57.864639 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:57.864682 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
W20260926 12:38:57.864831 17840 leader_election.cc:334] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019): Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:57.864919 17840 leader_election.cc:302] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 747538dff7b84fb6864644aacd55ff47; no voters: c19fed68d8e04971bcf0fb5fd081c5e4, fb639df2d8654bfabc75be8aeed5d93f
I20260926 12:38:57.865032 18563 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 12:38:58.336066 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:58.335615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:50577)
0926 12:38:58.335881 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:58.335946 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":116,"threads_started":1}
I20260926 12:38:58.344378 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:58.343964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:58189)
0926 12:38:58.344197 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:58.344255 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260926 12:38:58.344870 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:58.344552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.131:53953)
0926 12:38:58.344750 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:58.344799 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260926 12:38:59.337639 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.336990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:37899)
0926 12:38:59.337395 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.337502 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260926 12:38:59.345898 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.345320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:36071)
0926 12:38:59.345673 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.345764 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
I20260926 12:38:59.346098 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.345542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.131:44863)
0926 12:38:59.345881 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.345957 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20260926 12:38:59.416226 18578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:59.416443 18578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } }
I20260926 12:38:59.417095 18578 leader_election.cc:288] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019), fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188:35243)
I20260926 12:38:59.417795 18050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "747538dff7b84fb6864644aacd55ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb639df2d8654bfabc75be8aeed5d93f" is_pre_election: true
I20260926 12:38:59.417979 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.417457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:35164)
0926 12:38:59.417762 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.417855 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260926 12:38:59.417996 18050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747538dff7b84fb6864644aacd55ff47 in term 2.
I20260926 12:38:59.418435 17843 leader_election.cc:302] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747538dff7b84fb6864644aacd55ff47, fb639df2d8654bfabc75be8aeed5d93f; no voters:
I20260926 12:38:59.418702 18578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 12:38:59.418769 18578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:59.418809 18578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:38:59.418937 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.418685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:35170)
0926 12:38:59.418802 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.418866 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 12:38:59.419088 17840 leader_election.cc:334] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019): Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:59.420248 18578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } }
I20260926 12:38:59.420675 18578 leader_election.cc:288] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 3 election: Requested vote from peers c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019), fb639df2d8654bfabc75be8aeed5d93f (127.10.224.188:35243)
I20260926 12:38:59.421051 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.420783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:35172)
0926 12:38:59.420921 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.420972 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 12:38:59.421212 18050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "747538dff7b84fb6864644aacd55ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb639df2d8654bfabc75be8aeed5d93f"
I20260926 12:38:59.421500 18050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:38:59.421741 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.421532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:35176)
0926 12:38:59.421638 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.421683 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 12:38:59.421841 17840 leader_election.cc:334] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019): Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
I20260926 12:38:59.422724 18050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747538dff7b84fb6864644aacd55ff47 in term 3.
I20260926 12:38:59.423257 17843 leader_election.cc:302] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 747538dff7b84fb6864644aacd55ff47, fb639df2d8654bfabc75be8aeed5d93f; no voters: c19fed68d8e04971bcf0fb5fd081c5e4
I20260926 12:38:59.423617 18578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:38:59.423863 18578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [term 3 LEADER]: Becoming Leader. State: Replica: 747538dff7b84fb6864644aacd55ff47, State: Running, Role: LEADER
I20260926 12:38:59.424274 18578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } }
I20260926 12:38:59.425715 18581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 747538dff7b84fb6864644aacd55ff47. Latest consensus state: current_term: 3 leader_uuid: "747538dff7b84fb6864644aacd55ff47" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:59.425900 18581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:59.426280 18583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:59.429028 18583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:59.429458 18583 catalog_manager.cc:1259] Loaded cluster ID: ab7db6a15d1d43bcacdf3cea4b45ead3
I20260926 12:38:59.429522 18583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:59.430145 18583 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:59.430532 18583 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Loaded TSK: 0
I20260926 12:38:59.431406 18583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:38:59.474295 17854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:38:59.478075 17854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:38:59.482539 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.482070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:35196)
0926 12:38:59.482333 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.482430 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 12:38:59.482573 18050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [term 3 FOLLOWER]: Refusing update from remote peer 747538dff7b84fb6864644aacd55ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 12:38:59.483206 18581 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 12:38:59.483582 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.483272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.0.0.1:35200)
0926 12:38:59.483407 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.483479 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 12:38:59.483755 17840 consensus_peers.cc:487] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 -> Peer c19fed68d8e04971bcf0fb5fd081c5e4 (127.10.224.190:38019): Couldn't send request to peer c19fed68d8e04971bcf0fb5fd081c5e4. Status: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 12:38:59.485309 18585 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 747538dff7b84fb6864644aacd55ff47. Latest consensus state: current_term: 3 leader_uuid: "747538dff7b84fb6864644aacd55ff47" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:59.485507 18585 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:59.487006 18581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "747538dff7b84fb6864644aacd55ff47" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:59.487176 18581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:59.487408 18585 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "747538dff7b84fb6864644aacd55ff47" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c19fed68d8e04971bcf0fb5fd081c5e4" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38019 } } peers { permanent_uuid: "747538dff7b84fb6864644aacd55ff47" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 33525 } attrs { promote: false } } peers { permanent_uuid: "fb639df2d8654bfabc75be8aeed5d93f" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 35243 } attrs { promote: false } } }
I20260926 12:38:59.487583 18585 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb639df2d8654bfabc75be8aeed5d93f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:59.488080 18586 mvcc.cc:205] Tried to move back new op lower bound from 7333586286512521216 to 7333586286284935168. Current Snapshot: MvccSnapshot[applied={T|T < 7333586286512521216}]
I20260926 12:38:59.499847 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.494815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.10.224.130:41439)
0926 12:38:59.495165 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:59.495170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:59.495338 (+ 168us) server_negotiation.cc:407] Connection header received
0926 12:38:59.495963 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:59.495970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:59.496037 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:59.496148 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:59.496506 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.497248 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.498414 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.499085 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.499210 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.499267 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:59.499584 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 12:38:59.499671 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":143,"threads_started":1}
I20260926 12:38:59.500114 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.494930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44163 (local address 127.0.0.1:58554)
0926 12:38:59.495657 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.495674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:59.495788 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:59.496460 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:59.496465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:59.496483 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:59.496601 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.496607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.497517 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.497522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.498772 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.498781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.499596 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.499600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.499700 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.499739 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 12:38:59.499822 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:38:59.499916 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"spinlock_wait_cycles":2176,"thread_start_us":147,"threads_started":1}
I20260926 12:38:59.500602 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.495113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.10.224.129:44163)
0926 12:38:59.496097 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:59.496103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:59.496119 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:38:59.496179 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:59.496182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:59.496234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:59.496329 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:59.496733 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.497385 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.498930 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.499431 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.499635 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.499704 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:59.500326 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 12:38:59.500428 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":79,"threads_started":1}
I20260926 12:38:59.500919 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.494679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41439 (local address 127.0.0.1:35698)
0926 12:38:59.495233 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.495250 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:59.495431 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:59.496162 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:59.496165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:59.496184 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:59.496358 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.496366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.497380 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.497384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.498269 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.498277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.499332 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.499336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.499423 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.499448 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:38:59.499512 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:38:59.500767 (+ 1255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475}
I20260926 12:38:59.501138 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.495330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:33013 (local address 127.0.0.1:43434)
0926 12:38:59.495657 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.495674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:59.495788 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:59.496282 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:59.496285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:59.496300 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:59.496456 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.496465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.498467 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.498470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.499633 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.499640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.500705 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.500709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.500822 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.500846 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:38:59.500931 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 12:38:59.501009 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":1,"spinlock_wait_cycles":7424,"thread_start_us":96,"threads_started":1}
I20260926 12:38:59.501513 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.495492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.10.224.131:33013)
0926 12:38:59.495795 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:59.495811 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:59.495830 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:38:59.495901 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:59.495904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:59.495973 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:59.496087 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:59.497482 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.498361 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.499771 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.500488 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.500636 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.500711 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:59.501111 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 12:38:59.501218 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
I20260926 12:38:59.501636 18153 tablet_service.cc:1467] Processing CreateTablet for tablet 525774c590c94d50ab69eaf7141fb620 (DEFAULT_TABLE table=test-table [id=aba33d405aa1420aa156939f0da06d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:59.502031 18153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525774c590c94d50ab69eaf7141fb620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:59.502422 18426 tablet_service.cc:1467] Processing CreateTablet for tablet 525774c590c94d50ab69eaf7141fb620 (DEFAULT_TABLE table=test-table [id=aba33d405aa1420aa156939f0da06d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:59.502782 18426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525774c590c94d50ab69eaf7141fb620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:59.503921 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 525774c590c94d50ab69eaf7141fb620 (DEFAULT_TABLE table=test-table [id=aba33d405aa1420aa156939f0da06d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:38:59.504261 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525774c590c94d50ab69eaf7141fb620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:59.505404 18596 tablet_bootstrap.cc:492] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94: Bootstrap starting.
I20260926 12:38:59.505868 18597 tablet_bootstrap.cc:492] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: Bootstrap starting.
I20260926 12:38:59.506500 18596 tablet_bootstrap.cc:654] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:59.506731 18597 tablet_bootstrap.cc:654] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:59.507103 18598 tablet_bootstrap.cc:492] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: Bootstrap starting.
I20260926 12:38:59.508196 18596 tablet_bootstrap.cc:492] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94: No bootstrap required, opened a new log
I20260926 12:38:59.508244 18598 tablet_bootstrap.cc:654] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:59.508327 18596 ts_tablet_manager.cc:1397] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:38:59.508953 18597 tablet_bootstrap.cc:492] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: No bootstrap required, opened a new log
I20260926 12:38:59.508912 18596 raft_consensus.cc:348] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.509037 18597 ts_tablet_manager.cc:1397] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:38:59.509037 18596 raft_consensus.cc:374] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:59.509070 18596 raft_consensus.cc:729] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4631bcece0204ad2a8252fc5fe1a0d94, State: Initialized, Role: FOLLOWER
I20260926 12:38:59.509245 18596 consensus_queue.cc:260] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.509446 18597 raft_consensus.cc:348] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.509557 18597 raft_consensus.cc:374] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:59.509588 18597 raft_consensus.cc:729] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89039a18f4734d29ae5de038215d21aa, State: Initialized, Role: FOLLOWER
I20260926 12:38:59.509723 18596 ts_tablet_manager.cc:1428] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:59.509716 18597 consensus_queue.cc:260] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.510066 18597 ts_tablet_manager.cc:1428] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:59.510483 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.510122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:38297)
0926 12:38:59.510313 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.510381 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 12:38:59.511670 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.511343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:34373)
0926 12:38:59.511508 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.511590 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 12:38:59.512655 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.511094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.131:47627)
0926 12:38:59.512475 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.512556 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1256}
I20260926 12:38:59.513139 18598 tablet_bootstrap.cc:492] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: No bootstrap required, opened a new log
I20260926 12:38:59.513375 18598 ts_tablet_manager.cc:1397] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 12:38:59.514015 18598 raft_consensus.cc:348] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.514159 18598 raft_consensus.cc:374] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:38:59.514218 18598 raft_consensus.cc:729] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea61906bb2b54dfab8df4c3cae1b812a, State: Initialized, Role: FOLLOWER
I20260926 12:38:59.514407 18598 consensus_queue.cc:260] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.514601 18221 heartbeater.cc:503] Master 127.10.224.189:33525 was elected leader, sending a full tablet report...
I20260926 12:38:59.514840 18598 ts_tablet_manager.cc:1428] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:38:59.515087 18494 heartbeater.cc:503] Master 127.10.224.189:33525 was elected leader, sending a full tablet report...
I20260926 12:38:59.515580 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.515252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:38611)
0926 12:38:59.515378 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.515474 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 12:38:59.516391 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.516125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:57377)
0926 12:38:59.516275 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.516326 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 12:38:59.517709 18358 heartbeater.cc:503] Master 127.10.224.189:33525 was elected leader, sending a full tablet report...
I20260926 12:38:59.603185 18602 raft_consensus.cc:482] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:38:59.603447 18602 raft_consensus.cc:504] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.604877 18602 leader_election.cc:288] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439), 89039a18f4734d29ae5de038215d21aa (127.10.224.131:33013)
I20260926 12:38:59.610898 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.605287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:52339 (local address 127.10.224.130:41439)
0926 12:38:59.605450 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:59.605455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:59.605943 (+ 488us) server_negotiation.cc:407] Connection header received
0926 12:38:59.606123 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:59.606126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:59.606168 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:59.606236 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:59.606542 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.607290 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.609414 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.610058 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.610209 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.610268 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:59.610651 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 12:38:59.610740 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 12:38:59.610903 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.605428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:48269 (local address 127.10.224.131:33013)
0926 12:38:59.605674 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:59.605680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:59.605700 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:38:59.605785 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:59.605788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:59.605836 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:59.605965 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:38:59.606394 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.607221 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.609414 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.610069 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.610209 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.610268 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:59.610651 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 12:38:59.610740 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 12:38:59.610960 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.605245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:33013 (local address 127.10.224.129:48269)
0926 12:38:59.605505 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.605522 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:59.605659 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:59.606007 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:59.606012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:59.606033 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:59.606257 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.606264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.607385 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.607389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.609267 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.609273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.610234 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.610240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.610441 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.610533 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 12:38:59.610618 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 12:38:59.610725 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":255}
I20260926 12:38:59.610968 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.605293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41439 (local address 127.10.224.129:52339)
0926 12:38:59.605812 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.605826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:59.605989 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:59.606273 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:59.606275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:59.606295 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:38:59.606429 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.606434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.607406 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.607409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.609261 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:59.609273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.610243 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.610246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:59.610439 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.610529 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 12:38:59.610618 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:38:59.610725 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":166,"thread_start_us":188,"threads_started":1}
I20260926 12:38:59.611910 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525774c590c94d50ab69eaf7141fb620" candidate_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" is_pre_election: true
I20260926 12:38:59.612174 18310 raft_consensus.cc:2454] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4631bcece0204ad2a8252fc5fe1a0d94 in term 0.
I20260926 12:38:59.612459 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525774c590c94d50ab69eaf7141fb620" candidate_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89039a18f4734d29ae5de038215d21aa" is_pre_election: true
I20260926 12:38:59.612735 18109 leader_election.cc:302] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4631bcece0204ad2a8252fc5fe1a0d94, ea61906bb2b54dfab8df4c3cae1b812a; no voters:
I20260926 12:38:59.612789 18446 raft_consensus.cc:2454] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4631bcece0204ad2a8252fc5fe1a0d94 in term 0.
I20260926 12:38:59.613022 18602 raft_consensus.cc:2781] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:38:59.613107 18602 raft_consensus.cc:482] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:38:59.613169 18602 raft_consensus.cc:3037] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:59.614471 18602 raft_consensus.cc:504] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.615051 18602 leader_election.cc:288] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [CANDIDATE]: Term 1 election: Requested vote from peers ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439), 89039a18f4734d29ae5de038215d21aa (127.10.224.131:33013)
I20260926 12:38:59.615608 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525774c590c94d50ab69eaf7141fb620" candidate_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea61906bb2b54dfab8df4c3cae1b812a"
I20260926 12:38:59.615643 18446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525774c590c94d50ab69eaf7141fb620" candidate_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89039a18f4734d29ae5de038215d21aa"
I20260926 12:38:59.615778 18310 raft_consensus.cc:3037] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:59.615830 18446 raft_consensus.cc:3037] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:59.617084 18446 raft_consensus.cc:2454] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4631bcece0204ad2a8252fc5fe1a0d94 in term 1.
I20260926 12:38:59.617139 18310 raft_consensus.cc:2454] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4631bcece0204ad2a8252fc5fe1a0d94 in term 1.
I20260926 12:38:59.617535 18106 leader_election.cc:302] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4631bcece0204ad2a8252fc5fe1a0d94, 89039a18f4734d29ae5de038215d21aa; no voters:
I20260926 12:38:59.617792 18602 raft_consensus.cc:2781] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:59.618072 18602 raft_consensus.cc:686] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [term 1 LEADER]: Becoming Leader. State: Replica: 4631bcece0204ad2a8252fc5fe1a0d94, State: Running, Role: LEADER
I20260926 12:38:59.618466 18602 consensus_queue.cc:237] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } }
I20260926 12:38:59.618973 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.618651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:52445)
0926 12:38:59.618804 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.618873 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 12:38:59.622210 17855 catalog_manager.cc:5495] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4631bcece0204ad2a8252fc5fe1a0d94 (127.10.224.129). New cstate: current_term: 1 leader_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4631bcece0204ad2a8252fc5fe1a0d94" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 } health_report { overall_health: UNKNOWN } } }
I20260926 12:38:59.660445 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.656065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.10.224.129:44163)
0926 12:38:59.656246 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:59.656252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:59.656270 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:38:59.656460 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:59.656462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:59.656548 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:59.656758 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:59.657048 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.657648 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.658577 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:59.658827 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:59.658943 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:59.659263 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:59.659956 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:59.660099 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:59.660173 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 12:38:59.660248 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 12:38:59.664618 18310 raft_consensus.cc:1261] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 1 FOLLOWER]: Refusing update from remote peer 4631bcece0204ad2a8252fc5fe1a0d94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:59.664830 18446 raft_consensus.cc:1261] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 1 FOLLOWER]: Refusing update from remote peer 4631bcece0204ad2a8252fc5fe1a0d94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:59.665436 18602 consensus_queue.cc:1035] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea61906bb2b54dfab8df4c3cae1b812a" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:59.665668 18606 consensus_queue.cc:1035] T 525774c590c94d50ab69eaf7141fb620 P 4631bcece0204ad2a8252fc5fe1a0d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89039a18f4734d29ae5de038215d21aa" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:59.667874 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.667504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:42007)
0926 12:38:59.667718 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.667781 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 12:38:59.670091 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.669789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.131:40817)
0926 12:38:59.669910 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.670006 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 12:38:59.672977 18610 mvcc.cc:205] Tried to move back new op lower bound from 7333586287259385856 to 7333586287080988672. Current Snapshot: MvccSnapshot[applied={T|T < 7333586287259385856}]
I20260926 12:38:59.674401 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.673932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.130:41645)
0926 12:38:59.674243 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.674308 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
I20260926 12:38:59.676144 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.672912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.131:36183)
0926 12:38:59.675914 (+ 3002us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.676001 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2887}
I20260926 12:38:59.678795 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.673237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38019 (local address 127.10.224.129:54441)
0926 12:38:59.678588 (+ 5351us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.678684 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5258}
I20260926 12:38:59.684098 17855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:59.684322 17855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:38:59.689296 17855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 747538dff7b84fb6864644aacd55ff47: Sending DeleteTablet for 3 replicas of tablet 525774c590c94d50ab69eaf7141fb620
I20260926 12:38:59.690466 18290 tablet_service.cc:1514] Processing DeleteTablet for tablet 525774c590c94d50ab69eaf7141fb620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:59 UTC) from {username='oryx', principal='oryx/127.10.224.189@KRBTEST.COM'} at 127.0.0.1:35698
I20260926 12:38:59.691061 18619 tablet_replica.cc:330] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: stopping tablet replica
I20260926 12:38:59.691031 18426 tablet_service.cc:1514] Processing DeleteTablet for tablet 525774c590c94d50ab69eaf7141fb620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:59 UTC) from {username='oryx', principal='oryx/127.10.224.189@KRBTEST.COM'} at 127.0.0.1:43434
I20260926 12:38:59.691241 18619 raft_consensus.cc:2229] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:59.691488 18619 raft_consensus.cc:2258] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:59.691723 18153 tablet_service.cc:1514] Processing DeleteTablet for tablet 525774c590c94d50ab69eaf7141fb620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:38:59 UTC) from {username='oryx', principal='oryx/127.10.224.189@KRBTEST.COM'} at 127.0.0.1:58554
I20260926 12:38:59.692478 18620 tablet_replica.cc:330] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: stopping tablet replica
I20260926 12:38:59.692582 18619 ts_tablet_manager.cc:1905] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:59.692683 18620 raft_consensus.cc:2229] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:38:59.692881 18620 raft_consensus.cc:2258] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:38:59.693053 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 18000
I20260926 12:38:59.694211 18620 ts_tablet_manager.cc:1905] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:38:59.696630 18620 ts_tablet_manager.cc:1918] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:59.696728 18620 log.cc:1196] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/525774c590c94d50ab69eaf7141fb620
I20260926 12:38:59.697029 18620 ts_tablet_manager.cc:1939] T 525774c590c94d50ab69eaf7141fb620 P 89039a18f4734d29ae5de038215d21aa: Deleting consensus metadata
I20260926 12:38:59.697935 17840 catalog_manager.cc:4841] TS 89039a18f4734d29ae5de038215d21aa (127.10.224.131:33013): tablet 525774c590c94d50ab69eaf7141fb620 (table test-table [id=aba33d405aa1420aa156939f0da06d24]) successfully deleted
I20260926 12:38:59.699033 18619 ts_tablet_manager.cc:1918] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:38:59.699116 18619 log.cc:1196] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/525774c590c94d50ab69eaf7141fb620
I20260926 12:38:59.699395 18619 ts_tablet_manager.cc:1939] T 525774c590c94d50ab69eaf7141fb620 P ea61906bb2b54dfab8df4c3cae1b812a: Deleting consensus metadata
I20260926 12:38:59.700243 17843 catalog_manager.cc:4841] TS ea61906bb2b54dfab8df4c3cae1b812a (127.10.224.130:41439): tablet 525774c590c94d50ab69eaf7141fb620 (table test-table [id=aba33d405aa1420aa156939f0da06d24]) successfully deleted
I20260926 12:38:59.701710 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 18227
I20260926 12:38:59.702013 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:59.701732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44163 (local address 127.0.0.1:58576)
0926 12:38:59.701859 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:59.701917 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:38:59.702562 17840 catalog_manager.cc:4584] TS 4631bcece0204ad2a8252fc5fe1a0d94 (127.10.224.129:44163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 525774c590c94d50ab69eaf7141fb620: Network error: Client connection negotiation failed: client connection to 127.10.224.129:44163: connect: Connection refused (error 111)
I20260926 12:38:59.709250 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 18363
I20260926 12:38:59.716946 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 17719
I20260926 12:38:59.725036 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 17828
2026-09-26T12:38:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7368 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:38:59 dist-test-slave-m3pj krb5kdc[18624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:38:59 dist-test-slave-m3pj krb5kdc[18624](info): set up 2 sockets
Sep 26 12:38:59 dist-test-slave-m3pj krb5kdc[18624](info): commencing operation
krb5kdc: starting...
W20260926 12:39:00.793004 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:39:00 dist-test-slave-m3pj krb5kdc[18624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:39:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.190.keytab.
Entry for principal oryx/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.190.keytab.
I20260926 12:39:00.931828 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:36275
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.190.keytab
--principal=oryx/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:36275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:01.056835 18637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:01.057193 18637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:01.057238 18637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:01.061830 18637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:39:01.061931 18637 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:39:01.061950 18637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:01.061969 18637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:01.061986 18637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:39:01.062009 18637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:39:01.069367 18637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:36275
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.190.keytab
--principal=oryx/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:36275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:01.070729 18637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:01.072033 18637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:01.083622 18645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:01.083657 18643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:01.083691 18637 server_base.cc:1034] running on GCE node
W20260926 12:39:01.083626 18642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:01.084724 18637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:01.086541 18637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:01.087800 18637 hybrid_clock.cc:648] HybridClock initialized: now 1790426341087763 us; error 56 us; skew 500 ppm
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:01.091398 18637 init.cc:377] Logged in from keytab as oryx/127.10.224.190@KRBTEST.COM (short username oryx)
I20260926 12:39:01.092607 18637 webserver.cc:460] Webserver started at http://127.10.224.190:44193/ using document root <none> and password file <none>
I20260926 12:39:01.093253 18637 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:01.093353 18637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:01.093595 18637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:01.096046 18637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "bd00e4e691234b54b8742a40dfb0e62e"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "ec36aead1d4604368d860cab54708731"
server_key_iv: "a5dd2b7a3d71e0908375be4e39a52ead"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.096709 18637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "bd00e4e691234b54b8742a40dfb0e62e"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "ec36aead1d4604368d860cab54708731"
server_key_iv: "a5dd2b7a3d71e0908375be4e39a52ead"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.101292 18637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:39:01.104441 18652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:01.106221 18637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:01.106424 18637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "bd00e4e691234b54b8742a40dfb0e62e"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "ec36aead1d4604368d860cab54708731"
server_key_iv: "a5dd2b7a3d71e0908375be4e39a52ead"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.106578 18637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:01.121428 18637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:01.125114 18637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:01.125349 18637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:01.133896 18637 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:36275
I20260926 12:39:01.133948 18714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:36275 every 8 connection(s)
I20260926 12:39:01.134994 18637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:39:01.138191 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:01.139813 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 18637
I20260926 12:39:01.139986 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:39:01.140259 11138 external_mini_cluster.cc:1428] Setting key c61c8487376c2e1ca7ac26817e5aad1b
I20260926 12:39:01.145047 18715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: Bootstrap starting.
I20260926 12:39:01.147881 18715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:01.148866 18715 log.cc:824] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:39:01.151322 18715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: No bootstrap required, opened a new log
I20260926 12:39:01.154834 18715 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00e4e691234b54b8742a40dfb0e62e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36275 } }
I20260926 12:39:01.155129 18715 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:01.155206 18715 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd00e4e691234b54b8742a40dfb0e62e, State: Initialized, Role: FOLLOWER
I20260926 12:39:01.155747 18715 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00e4e691234b54b8742a40dfb0e62e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36275 } }
I20260926 12:39:01.155929 18715 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:39:01.156009 18715 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:39:01.156129 18715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:01.156502 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.141858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.10.224.190:36275)
0926 12:39:01.142290 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.142313 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.142350 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:39:01.143004 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.143022 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.143379 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.143754 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.144702 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.145790 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.147011 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.147638 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.147792 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.150976 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:01.150982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:01.150997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:01.151027 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:01.153082 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.153629 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.153635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.153641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.153770 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.154081 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.154086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.154088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.154551 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:01.154772 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:01.155129 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 12:39:01.155449 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
I20260926 12:39:01.157186 18715 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00e4e691234b54b8742a40dfb0e62e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36275 } }
I20260926 12:39:01.157572 18715 leader_election.cc:302] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd00e4e691234b54b8742a40dfb0e62e; no voters:
I20260926 12:39:01.157907 18715 leader_election.cc:288] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:39:01.158006 18720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:01.158232 18720 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [term 1 LEADER]: Becoming Leader. State: Replica: bd00e4e691234b54b8742a40dfb0e62e, State: Running, Role: LEADER
I20260926 12:39:01.158635 18720 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00e4e691234b54b8742a40dfb0e62e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36275 } }
I20260926 12:39:01.159236 18715 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:39:01.160239 18721 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd00e4e691234b54b8742a40dfb0e62e. Latest consensus state: current_term: 1 leader_uuid: "bd00e4e691234b54b8742a40dfb0e62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00e4e691234b54b8742a40dfb0e62e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36275 } } }
I20260926 12:39:01.160378 18721 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:01.160624 18722 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd00e4e691234b54b8742a40dfb0e62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00e4e691234b54b8742a40dfb0e62e" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36275 } } }
I20260926 12:39:01.160724 18722 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:01.160931 18729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:39:01.164590 18729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:39:01.170747 18729 catalog_manager.cc:1347] Generated new cluster ID: 23a59f066f5d4599abd3729c234ffe6a
I20260926 12:39:01.170835 18729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:39:01.179088 18729 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:39:01.180205 18729 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:39:01.197577 18729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: Generated new TSK 0
I20260926 12:39:01.198491 18729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab.
Entry for principal oryx/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab.
I20260926 12:39:01.242758 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:36275
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab
--principal=oryx/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:01.370666 18741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:01.371026 18741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:01.371079 18741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:01.375505 18741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:01.375635 18741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:01.375679 18741 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:39:01.375713 18741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:39:01.381593 18741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab
--principal=oryx/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:36275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:01.382822 18741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:01.383977 18741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:39:01.394893 18741 server_base.cc:1034] running on GCE node
W20260926 12:39:01.394860 18749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:01.394794 18747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:01.394790 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:01.395748 18741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:01.396442 18741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:01.397619 18741 hybrid_clock.cc:648] HybridClock initialized: now 1790426341397595 us; error 39 us; skew 500 ppm
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:01.400949 18741 init.cc:377] Logged in from keytab as oryx/127.10.224.129@KRBTEST.COM (short username oryx)
I20260926 12:39:01.401983 18741 webserver.cc:460] Webserver started at http://127.10.224.129:38495/ using document root <none> and password file <none>
I20260926 12:39:01.402590 18741 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:01.402657 18741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:01.402879 18741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:01.405100 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "1e6a5856e3b14ee9a95bf003bb4341b8"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "d0ba3680bf76688d554beb6170e695fc"
server_key_iv: "7479482846f7ed4ba8945c677352f5e1"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.405697 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "1e6a5856e3b14ee9a95bf003bb4341b8"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "d0ba3680bf76688d554beb6170e695fc"
server_key_iv: "7479482846f7ed4ba8945c677352f5e1"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.409921 18741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:39:01.412891 18756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:01.414420 18741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:01.414544 18741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "1e6a5856e3b14ee9a95bf003bb4341b8"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "d0ba3680bf76688d554beb6170e695fc"
server_key_iv: "7479482846f7ed4ba8945c677352f5e1"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.414711 18741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:01.445525 18741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:01.448854 18741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:01.449069 18741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:01.450645 18741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:01.450721 18741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:01.450791 18741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:01.450831 18741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:39:01.462459 18741 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:40957
I20260926 12:39:01.462491 18871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:40957 every 8 connection(s)
I20260926 12:39:01.463837 18741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:39:01.465725 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.451640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:38589 (local address 127.10.224.190:36275)
0926 12:39:01.451771 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.451775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.452925 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 12:39:01.454179 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.454183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.454253 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.454352 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.455597 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.456504 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.457672 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.458045 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.458162 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.461636 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:01.461641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:01.461644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:01.461672 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:01.463310 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.463975 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.463999 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.464001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.464090 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.464509 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.464513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.464514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.464774 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:01.464916 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:01.465409 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 12:39:01.465551 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:39:01.466570 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.452289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:36275 (local address 127.10.224.129:38589)
0926 12:39:01.452780 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:01.452822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:01.453919 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:01.454538 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:01.454545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:01.454900 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:01.455403 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.455417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.456623 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.456627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.457541 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.457552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.458220 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.458224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.458390 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.459087 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:01.459104 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:01.461410 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:01.463442 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.463449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.463464 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.463850 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.464187 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.464193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.464197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.464394 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.464890 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:01.464896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:01.465163 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 12:39:01.465349 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 12:39:01.465692 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":132,"threads_started":1}
I20260926 12:39:01.468000 18872 heartbeater.cc:347] Connected to a master server at 127.10.224.190:36275
I20260926 12:39:01.468216 18872 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:01.468747 18872 heartbeater.cc:511] Master 127.10.224.190:36275 requested a full tablet report, sending...
I20260926 12:39:01.470329 18669 ts_manager.cc:194] Registered new tserver with Master: 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957)
I20260926 12:39:01.471431 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 18741
I20260926 12:39:01.471621 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:39:01.471990 11138 external_mini_cluster.cc:1428] Setting key fa901caa955c42a77f61c14b5accbfd6
I20260926 12:39:01.471923 18669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:38589
I20260926 12:39:01.478610 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.470249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:38713 (local address 127.10.224.190:36275)
0926 12:39:01.470395 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.470401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.470524 (+ 123us) server_negotiation.cc:407] Connection header received
0926 12:39:01.470642 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.470645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.470689 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.470772 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.471177 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.471992 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.473108 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.473416 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.473493 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.475037 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:01.475040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:01.475043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:01.475065 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:01.476793 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.477195 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.477202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.477206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.477297 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.477601 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.477607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.477611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.477862 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:01.477967 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:01.478295 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 12:39:01.478424 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:39:01.478643 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.470174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:36275 (local address 127.10.224.129:38713)
0926 12:39:01.470445 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:01.470460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:01.470561 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:01.470803 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:01.470805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:01.470979 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:01.471076 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.471081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.472149 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.472152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.472967 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.472988 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.473652 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.473655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.473776 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.474361 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:01.474382 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:01.474892 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:01.476902 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.476905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.476907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.477103 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.477430 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.477432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.477434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.477514 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.477979 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:01.477982 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:01.478077 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 12:39:01.478240 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 12:39:01.478443 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab.
Entry for principal oryx/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab.
I20260926 12:39:01.502859 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:36275
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab
--principal=oryx/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:01.619385 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:01.619745 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:01.619796 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:01.624009 18881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:01.624085 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:01.624130 18881 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:39:01.624164 18881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:39:01.629868 18881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab
--principal=oryx/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:36275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:01.630960 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:01.632076 18881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:01.642350 18889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:01.642336 18887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:01.642540 18881 server_base.cc:1034] running on GCE node
W20260926 12:39:01.642395 18886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:01.643214 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:01.644057 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:01.645233 18881 hybrid_clock.cc:648] HybridClock initialized: now 1790426341645197 us; error 50 us; skew 500 ppm
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:01.648579 18881 init.cc:377] Logged in from keytab as oryx/127.10.224.130@KRBTEST.COM (short username oryx)
I20260926 12:39:01.649644 18881 webserver.cc:460] Webserver started at http://127.10.224.130:39451/ using document root <none> and password file <none>
I20260926 12:39:01.650331 18881 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:01.650434 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:01.650681 18881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:01.653010 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "891e5d429fd64062b7791ddf6584ee3b"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "b080f8061c0c1818515ba4ec2df3ac5e"
server_key_iv: "df1acc60d4a32b7730bdfabc55f9fe19"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.653692 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "891e5d429fd64062b7791ddf6584ee3b"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "b080f8061c0c1818515ba4ec2df3ac5e"
server_key_iv: "df1acc60d4a32b7730bdfabc55f9fe19"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.657975 18881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:39:01.660811 18896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:01.662149 18881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 12:39:01.662288 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "891e5d429fd64062b7791ddf6584ee3b"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "b080f8061c0c1818515ba4ec2df3ac5e"
server_key_iv: "df1acc60d4a32b7730bdfabc55f9fe19"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.662412 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:01.690323 18881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:01.694162 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:01.694402 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:01.696264 18881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:01.696341 18881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:01.696420 18881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:01.696470 18881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:39:01.708741 18881 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:41715
I20260926 12:39:01.708765 19011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:41715 every 8 connection(s)
I20260926 12:39:01.710201 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:39:01.712563 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.697099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:44799 (local address 127.10.224.190:36275)
0926 12:39:01.697262 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.697267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.698218 (+ 951us) server_negotiation.cc:407] Connection header received
0926 12:39:01.699227 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.699231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.699290 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.699388 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.700776 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.701535 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.703707 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.704108 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.704239 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.707926 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:01.707930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:01.707932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:01.707963 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:01.709888 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.710638 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.710646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.710651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.710771 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.711260 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.711264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.711266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.711512 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:01.711677 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:01.712271 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 12:39:01.712436 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:39:01.713408 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.697593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:36275 (local address 127.10.224.130:44799)
0926 12:39:01.698038 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:01.698097 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:01.699005 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:01.699648 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:01.699657 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:01.700104 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:01.700620 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.700635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.701698 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.701702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.703556 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.703574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.704318 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.704323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.704623 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.705456 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:01.705484 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:01.707662 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:01.710062 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.710070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.710083 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.710464 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.710902 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.710906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.710908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.711128 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.711682 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:01.711687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:01.711996 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 12:39:01.712165 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 12:39:01.712445 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260926 12:39:01.714931 19012 heartbeater.cc:347] Connected to a master server at 127.10.224.190:36275
I20260926 12:39:01.715131 19012 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:01.715806 19012 heartbeater.cc:511] Master 127.10.224.190:36275 requested a full tablet report, sending...
I20260926 12:39:01.716913 18669 ts_manager.cc:194] Registered new tserver with Master: 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715)
I20260926 12:39:01.717795 18669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:44799
I20260926 12:39:01.720184 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 18881
I20260926 12:39:01.720307 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:39:01.720628 11138 external_mini_cluster.cc:1428] Setting key 9aaad22c362632327b718ec607d98674
I20260926 12:39:01.725091 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.717315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:33167 (local address 127.10.224.190:36275)
0926 12:39:01.717456 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.717461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.717646 (+ 185us) server_negotiation.cc:407] Connection header received
0926 12:39:01.717779 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.717781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.717819 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.717885 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.718514 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.718999 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.720110 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.720454 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.720577 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.721808 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:01.721812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:01.721814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:01.721861 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:01.723295 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.723817 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.723823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.723825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.723907 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.724180 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.724185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.724186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.724414 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:01.724538 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:01.724796 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 12:39:01.724908 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:39:01.725109 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.717262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:36275 (local address 127.10.224.130:33167)
0926 12:39:01.717570 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:01.717586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:01.717693 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:01.717967 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:01.717970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:01.718187 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:01.718322 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.718330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.719129 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.719132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.719969 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.719979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.720602 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.720605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.720716 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.721305 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:01.721317 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:01.721673 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:01.723419 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.723422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.723424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.723708 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.724002 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.724005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.724007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.724094 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.724529 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:01.724532 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:01.724635 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 12:39:01.724741 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 12:39:01.724906 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for oryx/127.10.224.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab.
Entry for principal oryx/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab.
I20260926 12:39:01.752715 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:36275
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab
--principal=oryx/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:01.872301 19021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:01.872615 19021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:01.872663 19021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:01.877291 19021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:01.877414 19021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:01.877466 19021 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:39:01.877506 19021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:39:01.884086 19021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab
--principal=oryx/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.224.190:36275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:01.885366 19021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:01.886546 19021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:01.898067 19029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:01.898080 19027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:01.898080 19026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:01.898494 19021 server_base.cc:1034] running on GCE node
I20260926 12:39:01.899039 19021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:01.899820 19021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:01.901043 19021 hybrid_clock.cc:648] HybridClock initialized: now 1790426341901005 us; error 63 us; skew 500 ppm
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:01.904168 19021 init.cc:377] Logged in from keytab as oryx/127.10.224.131@KRBTEST.COM (short username oryx)
I20260926 12:39:01.905149 19021 webserver.cc:460] Webserver started at http://127.10.224.131:37717/ using document root <none> and password file <none>
I20260926 12:39:01.905715 19021 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:01.905778 19021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:01.905956 19021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:01.908303 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "529c76a0c0824948947993f0cabee993"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "93255e6071bfcad2b7fbdfe541bb64db"
server_key_iv: "d2eb257ad80098f61b1eb4263b57ec10"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.908886 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "529c76a0c0824948947993f0cabee993"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "93255e6071bfcad2b7fbdfe541bb64db"
server_key_iv: "d2eb257ad80098f61b1eb4263b57ec10"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.913247 19021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:39:01.916146 19036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:01.917546 19021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:01.917665 19021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "529c76a0c0824948947993f0cabee993"
format_stamp: "Formatted at 2026-09-26 12:39:01 on dist-test-slave-m3pj"
server_key: "93255e6071bfcad2b7fbdfe541bb64db"
server_key_iv: "d2eb257ad80098f61b1eb4263b57ec10"
server_key_version: "encryptionkey@0"
I20260926 12:39:01.917771 19021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:01.929039 19021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:01.932648 19021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:01.932822 19021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:01.934255 19021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:01.934338 19021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:01.934413 19021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:01.934461 19021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:39:01.946303 19021 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:39775
I20260926 12:39:01.946692 19151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:39775 every 8 connection(s)
I20260926 12:39:01.947464 19021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:39:01.950016 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 19021
I20260926 12:39:01.950147 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.935194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:47271 (local address 127.10.224.190:36275)
0926 12:39:01.935337 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.935341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.936536 (+ 1195us) server_negotiation.cc:407] Connection header received
0926 12:39:01.937718 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.937724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.937810 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.937878 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.939857 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.940430 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.942028 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.942483 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.942633 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.945988 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:01.945994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:01.945997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:01.946033 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:01.947743 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.948424 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.948428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.948430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.948541 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.948932 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.948935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.948936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.949131 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:01.949212 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:01.949856 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 12:39:01.949992 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:39:01.950176 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:39:01.950489 11138 external_mini_cluster.cc:1428] Setting key b90f744a5b95e0f89dd1f5cf6b914ef1
I20260926 12:39:01.951319 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.935851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:36275 (local address 127.10.224.131:47271)
0926 12:39:01.936372 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:01.936416 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:01.937446 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:01.938391 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:01.938401 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:01.938859 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:01.939679 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.939699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.940573 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.940577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.941843 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.941855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.942704 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.942709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.942961 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.943831 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:01.943848 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:01.945768 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:01.947878 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.947884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.947899 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.948282 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.948651 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.948656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.948659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.948832 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.949254 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:01.949260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:01.949540 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 12:39:01.949761 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 12:39:01.950213 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":133,"threads_started":1}
I20260926 12:39:01.954949 19152 heartbeater.cc:347] Connected to a master server at 127.10.224.190:36275
I20260926 12:39:01.955142 19152 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:01.955812 19152 heartbeater.cc:511] Master 127.10.224.190:36275 requested a full tablet report, sending...
I20260926 12:39:01.956990 18669 ts_manager.cc:194] Registered new tserver with Master: 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775)
I20260926 12:39:01.957597 18669 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:47271
I20260926 12:39:01.962921 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.955309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:38161 (local address 127.10.224.190:36275)
0926 12:39:01.955554 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.955561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.955655 (+ 94us) server_negotiation.cc:407] Connection header received
0926 12:39:01.955804 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.955807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.955865 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.955967 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.956509 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.957030 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.957916 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.958140 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.958252 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.959462 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:01.959465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:01.959468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:01.959588 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:01.961248 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.961714 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.961719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.961721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.961802 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:01.962086 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:01.962090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:01.962092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:01.962276 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:01.962381 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:01.962637 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 12:39:01.962750 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:39:01.962942 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.955210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:36275 (local address 127.10.224.131:38161)
0926 12:39:01.955558 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:01.955574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:01.955689 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:01.956043 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:01.956046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:01.956243 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:01.956354 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.956361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.957172 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.957176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.957797 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:01.957806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.958286 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.958289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:01.958367 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.958943 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:01.958955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:01.959317 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:01.961372 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.961375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.961378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.961609 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.961894 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:01.961897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:01.961899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:01.961996 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:01.962370 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:01.962373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:01.962474 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 12:39:01.962580 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 12:39:01.962745 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":167,"threads_started":1}
I20260926 12:39:01.964920 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:39:01.991060 18718 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:01.984233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.10.224.190:36275)
0926 12:39:01.984365 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.984369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.984440 (+ 71us) server_negotiation.cc:407] Connection header received
0926 12:39:01.984483 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.984485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.984527 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.984619 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.985389 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.985918 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.986766 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.986983 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.987088 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.990894 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39704: BlockingRecv error: recv got EOF from 127.0.0.1:39704 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 12:39:01 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:39:02.002278 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:01.992509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.10.224.190:36275)
0926 12:39:01.992660 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:01.992664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:01.992683 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:39:01.992894 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:01.992898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:01.992955 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:01.993036 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:01.993937 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.994552 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.995495 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:01.995798 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:01.995937 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:01.998919 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:01.998923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:01.998925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:01.998963 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:02.000552 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:02.001052 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:02.001055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:02.001056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:02.001148 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:02.001460 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:02.001463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:02.001464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:02.001706 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:02.001819 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.002043 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.002150 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:39:02.005796 18669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:39:02.009053 18669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:39:02.027817 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.020961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.10.224.129:40957)
0926 12:39:02.021255 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.021262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.021834 (+ 572us) server_negotiation.cc:407] Connection header received
0926 12:39:02.022097 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.022130 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.022437 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.023063 (+ 626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.023190 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.024596 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.026103 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.026888 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.027025 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.027164 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.027460 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.027654 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
I20260926 12:39:02.027890 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.021386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.10.224.131:39775)
0926 12:39:02.021731 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.021744 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.022247 (+ 503us) server_negotiation.cc:407] Connection header received
0926 12:39:02.022389 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.022394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.022586 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.023123 (+ 537us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.023255 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.024419 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.026034 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.026629 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.026797 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.026958 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.027645 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.027749 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":117,"threads_started":1}
I20260926 12:39:02.028638 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.020770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:39775 (local address 127.0.0.1:51568)
0926 12:39:02.022142 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.022156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.022265 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.022830 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.022833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.022853 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.022968 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.022975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.024571 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.024574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.025896 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.025905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.027028 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.027032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.027243 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.027288 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.027404 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.028451 (+ 1047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1312,"mutex_wait_us":104,"thread_start_us":97,"threads_started":1}
I20260926 12:39:02.028640 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.020602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40957 (local address 127.0.0.1:40562)
0926 12:39:02.021530 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.021590 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.021883 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.022662 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.022670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.022698 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.022891 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.022902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.024746 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.024750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.025956 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.025967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.027021 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.027025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.027259 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.027290 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.027420 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.028459 (+ 1039us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":968,"thread_start_us":88,"threads_started":1}
I20260926 12:39:02.029009 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.022398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41715 (local address 127.0.0.1:37170)
0926 12:39:02.022770 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.022785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.022868 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.023486 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.023489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.023558 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.023694 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.023703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.025482 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.025485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.026597 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.026606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.027952 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.027956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.028136 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.028689 (+ 553us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.028775 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.028860 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":507,"thread_start_us":150,"threads_started":1}
I20260926 12:39:02.029490 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.022616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.10.224.130:41715)
0926 12:39:02.023029 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.023044 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.023068 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:39:02.023148 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.023153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.023331 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.023478 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.024037 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.025356 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.026740 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.027602 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.027751 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.027884 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.029233 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.029344 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":111,"threads_started":1}
I20260926 12:39:02.031863 18806 tablet_service.cc:1467] Processing CreateTablet for tablet e1bf0113a6834c088dcb7163cb5ee161 (DEFAULT_TABLE table=test-table [id=d56f20e7604e47cf901347f563c25850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.031874 19086 tablet_service.cc:1467] Processing CreateTablet for tablet e1bf0113a6834c088dcb7163cb5ee161 (DEFAULT_TABLE table=test-table [id=d56f20e7604e47cf901347f563c25850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.032131 18946 tablet_service.cc:1467] Processing CreateTablet for tablet e1bf0113a6834c088dcb7163cb5ee161 (DEFAULT_TABLE table=test-table [id=d56f20e7604e47cf901347f563c25850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.032999 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bf0113a6834c088dcb7163cb5ee161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.033020 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bf0113a6834c088dcb7163cb5ee161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.033020 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bf0113a6834c088dcb7163cb5ee161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.042156 19180 tablet_bootstrap.cc:492] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: Bootstrap starting.
I20260926 12:39:02.042328 19181 tablet_bootstrap.cc:492] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: Bootstrap starting.
I20260926 12:39:02.043990 19182 tablet_bootstrap.cc:492] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: Bootstrap starting.
I20260926 12:39:02.044435 19181 tablet_bootstrap.cc:654] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.044668 19180 tablet_bootstrap.cc:654] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.045280 19181 log.cc:824] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:02.045626 19180 log.cc:824] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:02.046783 19182 tablet_bootstrap.cc:654] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.047785 19180 tablet_bootstrap.cc:492] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: No bootstrap required, opened a new log
I20260926 12:39:02.047792 19181 tablet_bootstrap.cc:492] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: No bootstrap required, opened a new log
I20260926 12:39:02.047803 19182 log.cc:824] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:02.048018 19181 ts_tablet_manager.cc:1397] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:39:02.048051 19180 ts_tablet_manager.cc:1397] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:39:02.050370 19182 tablet_bootstrap.cc:492] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: No bootstrap required, opened a new log
I20260926 12:39:02.050618 19182 ts_tablet_manager.cc:1397] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 12:39:02.052205 19180 raft_consensus.cc:348] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.052493 19180 raft_consensus.cc:374] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.052567 19180 raft_consensus.cc:729] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6a5856e3b14ee9a95bf003bb4341b8, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.052511 19181 raft_consensus.cc:348] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.052888 19181 raft_consensus.cc:374] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.052994 19181 raft_consensus.cc:729] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529c76a0c0824948947993f0cabee993, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.053094 19180 consensus_queue.cc:260] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.053455 19181 consensus_queue.cc:260] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.053853 18872 heartbeater.cc:503] Master 127.10.224.190:36275 was elected leader, sending a full tablet report...
I20260926 12:39:02.054185 19180 ts_tablet_manager.cc:1428] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:39:02.054631 19181 ts_tablet_manager.cc:1428] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:39:02.055003 19152 heartbeater.cc:503] Master 127.10.224.190:36275 was elected leader, sending a full tablet report...
I20260926 12:39:02.055303 19182 raft_consensus.cc:348] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.055651 19182 raft_consensus.cc:374] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.055713 19182 raft_consensus.cc:729] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891e5d429fd64062b7791ddf6584ee3b, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.056252 19182 consensus_queue.cc:260] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.056545 19188 raft_consensus.cc:482] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:39:02.056720 19188 raft_consensus.cc:504] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.057492 19182 ts_tablet_manager.cc:1428] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 12:39:02.057870 19012 heartbeater.cc:503] Master 127.10.224.190:36275 was elected leader, sending a full tablet report...
I20260926 12:39:02.059772 19188 leader_election.cc:288] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775), 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715)
I20260926 12:39:02.065151 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.059043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:39103 (local address 127.10.224.131:39775)
0926 12:39:02.059411 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.059416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.060298 (+ 882us) server_negotiation.cc:407] Connection header received
0926 12:39:02.060420 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.060426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.060512 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.060620 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.061057 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.061880 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.063631 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.064220 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.064375 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.064438 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.064689 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.064920 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260926 12:39:02.065434 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.059564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:60283 (local address 127.10.224.130:41715)
0926 12:39:02.059694 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.059701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.060673 (+ 972us) server_negotiation.cc:407] Connection header received
0926 12:39:02.060925 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.060929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.060989 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.061083 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.061682 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.062422 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.063920 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.064528 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.064645 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.064694 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.065187 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.065294 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:39:02.065480 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.059026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41715 (local address 127.10.224.129:60283)
0926 12:39:02.060554 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.060569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.060755 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.061347 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.061352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.061374 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.061511 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.061518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.062550 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.062554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.063788 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.063796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.064636 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.064642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.064746 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.064926 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.065088 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.065184 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"thread_start_us":101,"threads_started":1}
I20260926 12:39:02.065649 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.058341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:39775 (local address 127.10.224.129:39103)
0926 12:39:02.060062 (+ 1721us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.060079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.060263 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.060666 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.060669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.060693 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.060887 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.060909 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.062026 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.062032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.063428 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.063439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.064378 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.064382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.064518 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.064553 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.065358 (+ 805us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.065436 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1647,"mutex_wait_us":16,"spinlock_wait_cycles":1408,"thread_start_us":132,"threads_started":1}
I20260926 12:39:02.066200 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bf0113a6834c088dcb7163cb5ee161" candidate_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891e5d429fd64062b7791ddf6584ee3b" is_pre_election: true
I20260926 12:39:02.066259 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bf0113a6834c088dcb7163cb5ee161" candidate_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529c76a0c0824948947993f0cabee993" is_pre_election: true
I20260926 12:39:02.066577 19106 raft_consensus.cc:2454] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e6a5856e3b14ee9a95bf003bb4341b8 in term 0.
I20260926 12:39:02.066577 18966 raft_consensus.cc:2454] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e6a5856e3b14ee9a95bf003bb4341b8 in term 0.
I20260926 12:39:02.067230 18760 leader_election.cc:302] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6a5856e3b14ee9a95bf003bb4341b8, 529c76a0c0824948947993f0cabee993; no voters:
I20260926 12:39:02.067564 19186 raft_consensus.cc:2781] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:39:02.067677 19186 raft_consensus.cc:482] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:39:02.067729 19186 raft_consensus.cc:3037] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.068945 19186 raft_consensus.cc:504] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.069375 19186 leader_election.cc:288] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [CANDIDATE]: Term 1 election: Requested vote from peers 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775), 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715)
I20260926 12:39:02.069914 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bf0113a6834c088dcb7163cb5ee161" candidate_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529c76a0c0824948947993f0cabee993"
I20260926 12:39:02.069974 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bf0113a6834c088dcb7163cb5ee161" candidate_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891e5d429fd64062b7791ddf6584ee3b"
I20260926 12:39:02.070045 19106 raft_consensus.cc:3037] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.070153 18966 raft_consensus.cc:3037] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.071162 19106 raft_consensus.cc:2454] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e6a5856e3b14ee9a95bf003bb4341b8 in term 1.
I20260926 12:39:02.071605 18760 leader_election.cc:302] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6a5856e3b14ee9a95bf003bb4341b8, 529c76a0c0824948947993f0cabee993; no voters:
I20260926 12:39:02.071841 18966 raft_consensus.cc:2454] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e6a5856e3b14ee9a95bf003bb4341b8 in term 1.
I20260926 12:39:02.071913 19186 raft_consensus.cc:2781] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:02.072072 19186 raft_consensus.cc:686] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1e6a5856e3b14ee9a95bf003bb4341b8, State: Running, Role: LEADER
I20260926 12:39:02.072454 19186 consensus_queue.cc:237] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } }
I20260926 12:39:02.076331 18668 catalog_manager.cc:5495] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129). New cstate: current_term: 1 leader_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } health_report { overall_health: UNKNOWN } } }
I20260926 12:39:02.098928 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.094993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.10.224.129:40957)
0926 12:39:02.095160 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.095165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.095179 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:39:02.095261 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.095264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.095324 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.095430 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:39:02.095686 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.096337 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.097197 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.097437 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.097534 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.097808 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.098356 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.098460 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.098639 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.098746 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 12:39:02.105861 18966 raft_consensus.cc:1261] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Refusing update from remote peer 1e6a5856e3b14ee9a95bf003bb4341b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:02.106082 19106 raft_consensus.cc:1261] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Refusing update from remote peer 1e6a5856e3b14ee9a95bf003bb4341b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:02.106719 19188 consensus_queue.cc:1035] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:02.106971 19186 consensus_queue.cc:1035] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:02.127310 18667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:02.127591 18667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:02.130739 18667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: Sending DeleteTablet for 3 replicas of tablet e1bf0113a6834c088dcb7163cb5ee161
I20260926 12:39:02.131896 18946 tablet_service.cc:1514] Processing DeleteTablet for tablet e1bf0113a6834c088dcb7163cb5ee161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:37170
I20260926 12:39:02.131891 18806 tablet_service.cc:1514] Processing DeleteTablet for tablet e1bf0113a6834c088dcb7163cb5ee161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:40562
I20260926 12:39:02.132709 19204 tablet_replica.cc:330] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: stopping tablet replica
I20260926 12:39:02.132602 18667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:39:02.132649 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet e1bf0113a6834c088dcb7163cb5ee161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:51568
W20260926 12:39:02.133075 18667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:39:02.133126 19204 raft_consensus.cc:2229] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:02.133411 19206 tablet_replica.cc:330] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: stopping tablet replica
I20260926 12:39:02.133487 19204 raft_consensus.cc:2258] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.133833 19206 raft_consensus.cc:2229] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:02.134188 19206 raft_consensus.cc:2258] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.134922 19205 tablet_replica.cc:330] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: stopping tablet replica
I20260926 12:39:02.135213 19205 raft_consensus.cc:2229] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:39:02.135612 19205 raft_consensus.cc:2258] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.137009 19206 ts_tablet_manager.cc:1905] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.139910 19206 ts_tablet_manager.cc:1918] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:39:02.140025 19206 log.cc:1196] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/e1bf0113a6834c088dcb7163cb5ee161
I20260926 12:39:02.140350 19206 ts_tablet_manager.cc:1939] T e1bf0113a6834c088dcb7163cb5ee161 P 529c76a0c0824948947993f0cabee993: Deleting consensus metadata
I20260926 12:39:02.141727 18656 catalog_manager.cc:4841] TS 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775): tablet e1bf0113a6834c088dcb7163cb5ee161 (table test-table [id=d56f20e7604e47cf901347f563c25850]) successfully deleted
I20260926 12:39:02.142657 19204 ts_tablet_manager.cc:1905] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.143743 18806 tablet_service.cc:1467] Processing CreateTablet for tablet ecee785858eb4b89b93b10722844c569 (DEFAULT_TABLE table=test-table [id=9a91a0d2be464f8e816bef44b32c826c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.144086 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecee785858eb4b89b93b10722844c569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.144035 18946 tablet_service.cc:1467] Processing CreateTablet for tablet ecee785858eb4b89b93b10722844c569 (DEFAULT_TABLE table=test-table [id=9a91a0d2be464f8e816bef44b32c826c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.144296 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecee785858eb4b89b93b10722844c569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.145601 19204 ts_tablet_manager.cc:1918] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:39:02.145696 19204 log.cc:1196] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/e1bf0113a6834c088dcb7163cb5ee161
I20260926 12:39:02.146003 19204 ts_tablet_manager.cc:1939] T e1bf0113a6834c088dcb7163cb5ee161 P 891e5d429fd64062b7791ddf6584ee3b: Deleting consensus metadata
I20260926 12:39:02.146857 19182 tablet_bootstrap.cc:492] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: Bootstrap starting.
I20260926 12:39:02.147294 19180 tablet_bootstrap.cc:492] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: Bootstrap starting.
I20260926 12:39:02.148000 19182 tablet_bootstrap.cc:654] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.148272 19180 tablet_bootstrap.cc:654] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.149102 19086 tablet_service.cc:1467] Processing CreateTablet for tablet ecee785858eb4b89b93b10722844c569 (DEFAULT_TABLE table=test-table [id=9a91a0d2be464f8e816bef44b32c826c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.149436 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecee785858eb4b89b93b10722844c569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.149855 19182 tablet_bootstrap.cc:492] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: No bootstrap required, opened a new log
I20260926 12:39:02.149876 19180 tablet_bootstrap.cc:492] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: No bootstrap required, opened a new log
I20260926 12:39:02.149935 19182 ts_tablet_manager.cc:1397] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:02.149957 19180 ts_tablet_manager.cc:1397] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:02.150408 19182 raft_consensus.cc:348] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.150432 19180 raft_consensus.cc:348] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.150537 19182 raft_consensus.cc:374] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.150570 19182 raft_consensus.cc:729] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891e5d429fd64062b7791ddf6584ee3b, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.150574 19180 raft_consensus.cc:374] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.150620 19180 raft_consensus.cc:729] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6a5856e3b14ee9a95bf003bb4341b8, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.150696 19182 consensus_queue.cc:260] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.150765 19180 consensus_queue.cc:260] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.150946 19182 ts_tablet_manager.cc:1428] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:39:02.151067 19180 ts_tablet_manager.cc:1428] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:39:02.152452 19205 ts_tablet_manager.cc:1905] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.152546 18656 catalog_manager.cc:4841] TS 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715): tablet e1bf0113a6834c088dcb7163cb5ee161 (table test-table [id=d56f20e7604e47cf901347f563c25850]) successfully deleted
I20260926 12:39:02.153127 19181 tablet_bootstrap.cc:492] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: Bootstrap starting.
I20260926 12:39:02.154297 19181 tablet_bootstrap.cc:654] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.157523 19181 tablet_bootstrap.cc:492] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: No bootstrap required, opened a new log
I20260926 12:39:02.157681 19181 ts_tablet_manager.cc:1397] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 12:39:02.158246 19181 raft_consensus.cc:348] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.158385 19181 raft_consensus.cc:374] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.158433 19181 raft_consensus.cc:729] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529c76a0c0824948947993f0cabee993, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.158579 19181 consensus_queue.cc:260] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.159312 19205 ts_tablet_manager.cc:1918] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:39:02.159493 19205 log.cc:1196] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/e1bf0113a6834c088dcb7163cb5ee161
I20260926 12:39:02.160010 19205 ts_tablet_manager.cc:1939] T e1bf0113a6834c088dcb7163cb5ee161 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting consensus metadata
I20260926 12:39:02.161716 18655 catalog_manager.cc:4841] TS 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957): tablet e1bf0113a6834c088dcb7163cb5ee161 (table test-table [id=d56f20e7604e47cf901347f563c25850]) successfully deleted
I20260926 12:39:02.161793 19181 ts_tablet_manager.cc:1428] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20260926 12:39:02.198637 19153 tablet.cc:2369] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:39:02.211467 19013 tablet.cc:2369] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:39:02.215423 18873 tablet.cc:2369] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:39:02.332247 19187 raft_consensus.cc:482] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:39:02.332430 19187 raft_consensus.cc:504] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.333676 19187 leader_election.cc:288] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957), 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715)
I20260926 12:39:02.338187 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.334027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:47813 (local address 127.10.224.129:40957)
0926 12:39:02.334157 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.334162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.334264 (+ 102us) server_negotiation.cc:407] Connection header received
0926 12:39:02.334603 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.334607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.334666 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.334737 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.335126 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.335841 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.337116 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.337528 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.337655 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.337721 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.337961 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.338045 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:39:02.338227 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.334085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:36993 (local address 127.10.224.130:41715)
0926 12:39:02.334225 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.334232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.334450 (+ 218us) server_negotiation.cc:407] Connection header received
0926 12:39:02.334541 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.334544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.334602 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.334696 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.335105 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.335846 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.337071 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.337546 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.337693 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.337763 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.338008 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.338101 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:39:02.338191 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.333915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40957 (local address 127.10.224.131:47813)
0926 12:39:02.334132 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.334151 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.334271 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.334841 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.334850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.334870 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.334990 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.334997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.335965 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.335973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.336950 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.336961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.337665 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.337669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.337788 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.337843 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.337908 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.338028 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":49}
I20260926 12:39:02.338215 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.333992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41715 (local address 127.10.224.131:36993)
0926 12:39:02.334336 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.334351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.334454 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.334746 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.334752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.334774 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.334928 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.334936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.335982 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.335990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.336921 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.336932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.337713 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.337717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.337806 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.337867 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.337968 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.338045 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":76,"thread_start_us":101,"threads_started":1}
I20260926 12:39:02.338693 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecee785858eb4b89b93b10722844c569" candidate_uuid: "529c76a0c0824948947993f0cabee993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891e5d429fd64062b7791ddf6584ee3b" is_pre_election: true
I20260926 12:39:02.338868 18966 raft_consensus.cc:2454] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529c76a0c0824948947993f0cabee993 in term 0.
I20260926 12:39:02.338901 18826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecee785858eb4b89b93b10722844c569" candidate_uuid: "529c76a0c0824948947993f0cabee993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" is_pre_election: true
I20260926 12:39:02.339159 18826 raft_consensus.cc:2454] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529c76a0c0824948947993f0cabee993 in term 0.
I20260926 12:39:02.339298 19040 leader_election.cc:302] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529c76a0c0824948947993f0cabee993, 891e5d429fd64062b7791ddf6584ee3b; no voters:
I20260926 12:39:02.339637 19187 raft_consensus.cc:2781] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:39:02.339725 19187 raft_consensus.cc:482] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:39:02.339771 19187 raft_consensus.cc:3037] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.341028 19187 raft_consensus.cc:504] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.341531 19187 leader_election.cc:288] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [CANDIDATE]: Term 1 election: Requested vote from peers 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957), 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715)
I20260926 12:39:02.342082 18826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecee785858eb4b89b93b10722844c569" candidate_uuid: "529c76a0c0824948947993f0cabee993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8"
I20260926 12:39:02.342154 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecee785858eb4b89b93b10722844c569" candidate_uuid: "529c76a0c0824948947993f0cabee993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891e5d429fd64062b7791ddf6584ee3b"
I20260926 12:39:02.342288 18826 raft_consensus.cc:3037] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.342308 18966 raft_consensus.cc:3037] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.343412 18966 raft_consensus.cc:2454] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529c76a0c0824948947993f0cabee993 in term 1.
I20260926 12:39:02.343421 18826 raft_consensus.cc:2454] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529c76a0c0824948947993f0cabee993 in term 1.
I20260926 12:39:02.343863 19039 leader_election.cc:302] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6a5856e3b14ee9a95bf003bb4341b8, 529c76a0c0824948947993f0cabee993; no voters:
I20260926 12:39:02.344115 19187 raft_consensus.cc:2781] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:02.344352 19187 raft_consensus.cc:686] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 1 LEADER]: Becoming Leader. State: Replica: 529c76a0c0824948947993f0cabee993, State: Running, Role: LEADER
I20260926 12:39:02.344655 19187 consensus_queue.cc:237] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.348246 18667 catalog_manager.cc:5495] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 reported cstate change: term changed from 0 to 1, leader changed from <none> to 529c76a0c0824948947993f0cabee993 (127.10.224.131). New cstate: current_term: 1 leader_uuid: "529c76a0c0824948947993f0cabee993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } health_report { overall_health: HEALTHY } } }
Sep 26 12:39:02 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426341, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.190@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:39:02.414543 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.405499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:36275 (local address 127.0.0.1:39734)
0926 12:39:02.405857 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.405870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.406015 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.406257 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.406260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.406732 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:02.406833 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.406839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.407657 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.407662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.408209 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.408217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.408739 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.408743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.408814 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.409440 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:02.409454 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:02.410976 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:02.412900 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:02.412914 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:02.412923 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:02.413178 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:02.413460 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:02.413463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:02.413465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:02.413547 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:02.413937 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:02.413946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:02.414060 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.414172 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.414388 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260926 12:39:02.414788 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.405565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.10.224.190:36275)
0926 12:39:02.405784 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.405790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.405954 (+ 164us) server_negotiation.cc:407] Connection header received
0926 12:39:02.406126 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.406128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.406163 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.406240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:02.406957 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.407474 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.408411 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.408627 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.408735 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.411132 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:02.411136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:02.411138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:02.411159 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:02.412752 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:02.413278 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:02.413283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:02.413287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:02.413366 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:02.413636 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:02.413639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:02.413641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:02.413829 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:02.413938 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.414226 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.414581 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":74}
I20260926 12:39:02.419412 18667 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:39734:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:39:02.420244 18667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:39:02.425899 18946 tablet_service.cc:1467] Processing CreateTablet for tablet 35cce78f369047879b2b6efe212b5cd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3e29aab7d28451c9f9f768e92604a80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:39:02.425899 19086 tablet_service.cc:1467] Processing CreateTablet for tablet 35cce78f369047879b2b6efe212b5cd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3e29aab7d28451c9f9f768e92604a80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:39:02.425899 18806 tablet_service.cc:1467] Processing CreateTablet for tablet 35cce78f369047879b2b6efe212b5cd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3e29aab7d28451c9f9f768e92604a80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:39:02.426290 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cce78f369047879b2b6efe212b5cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.426313 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cce78f369047879b2b6efe212b5cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.426313 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cce78f369047879b2b6efe212b5cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.429450 19180 tablet_bootstrap.cc:492] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: Bootstrap starting.
I20260926 12:39:02.429453 19181 tablet_bootstrap.cc:492] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: Bootstrap starting.
I20260926 12:39:02.429450 19182 tablet_bootstrap.cc:492] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: Bootstrap starting.
I20260926 12:39:02.430455 19182 tablet_bootstrap.cc:654] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.430455 19180 tablet_bootstrap.cc:654] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.430799 19181 tablet_bootstrap.cc:654] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.431872 19180 tablet_bootstrap.cc:492] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: No bootstrap required, opened a new log
I20260926 12:39:02.431978 19180 ts_tablet_manager.cc:1397] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:02.432161 19182 tablet_bootstrap.cc:492] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: No bootstrap required, opened a new log
I20260926 12:39:02.432222 19181 tablet_bootstrap.cc:492] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: No bootstrap required, opened a new log
I20260926 12:39:02.432252 19182 ts_tablet_manager.cc:1397] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:02.432322 19181 ts_tablet_manager.cc:1397] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:02.432494 19180 raft_consensus.cc:348] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.432626 19180 raft_consensus.cc:374] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.432670 19180 raft_consensus.cc:729] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6a5856e3b14ee9a95bf003bb4341b8, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.432682 19182 raft_consensus.cc:348] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.432790 19182 raft_consensus.cc:374] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.432823 19182 raft_consensus.cc:729] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891e5d429fd64062b7791ddf6584ee3b, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.432818 19180 consensus_queue.cc:260] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.432835 19181 raft_consensus.cc:348] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.432941 19182 consensus_queue.cc:260] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.432971 19181 raft_consensus.cc:374] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.433033 19181 raft_consensus.cc:729] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529c76a0c0824948947993f0cabee993, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.433228 19182 ts_tablet_manager.cc:1428] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:39:02.433188 19181 consensus_queue.cc:260] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.433365 19188 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.433527 19188 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:39:02.433475 19189 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.433599 19189 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:39:02.433940 19187 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.434090 19187 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:39:02.434157 19180 ts_tablet_manager.cc:1428] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 12:39:02.435114 19181 ts_tablet_manager.cc:1428] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:02.455840 19189 raft_consensus.cc:482] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:39:02.456018 19189 raft_consensus.cc:504] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.457320 19189 leader_election.cc:288] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957), 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775)
I20260926 12:39:02.461872 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.457618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:34027 (local address 127.10.224.129:40957)
0926 12:39:02.457785 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.457791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.457933 (+ 142us) server_negotiation.cc:407] Connection header received
0926 12:39:02.458089 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.458092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.458161 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.458239 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.458576 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.459271 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.460777 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.461235 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.461351 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.461393 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.461634 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.461731 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 12:39:02.461979 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.457589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:39775 (local address 127.10.224.130:54493)
0926 12:39:02.457955 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.457972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.458100 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.458496 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.458499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.458547 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.458651 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.458658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.459642 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.459645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.460664 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.460674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.461545 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.461547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.461657 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.461679 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.461753 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.461821 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":56,"thread_start_us":128,"threads_started":1}
I20260926 12:39:02.461968 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.457729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:54493 (local address 127.10.224.131:39775)
0926 12:39:02.457850 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.457857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.458083 (+ 226us) server_negotiation.cc:407] Connection header received
0926 12:39:02.458249 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.458252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.458308 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.458395 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:02.458777 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.459505 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.460836 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.461422 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.461561 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.461626 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.461783 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.461855 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:39:02.462002 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.457487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40957 (local address 127.10.224.130:34027)
0926 12:39:02.457831 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.457848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.457968 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.458265 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.458268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.458305 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:02.458442 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.458450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.459373 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.459377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.460653 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.460664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.461366 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.461371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.461483 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.461525 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.461607 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.461701 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":21,"thread_start_us":88,"threads_started":1}
I20260926 12:39:02.462453 18826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cce78f369047879b2b6efe212b5cd8" candidate_uuid: "891e5d429fd64062b7791ddf6584ee3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" is_pre_election: true
I20260926 12:39:02.462666 18826 raft_consensus.cc:2454] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 891e5d429fd64062b7791ddf6584ee3b in term 0.
I20260926 12:39:02.462913 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cce78f369047879b2b6efe212b5cd8" candidate_uuid: "891e5d429fd64062b7791ddf6584ee3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529c76a0c0824948947993f0cabee993" is_pre_election: true
I20260926 12:39:02.463059 19106 raft_consensus.cc:2454] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 891e5d429fd64062b7791ddf6584ee3b in term 0.
I20260926 12:39:02.463085 18899 leader_election.cc:302] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6a5856e3b14ee9a95bf003bb4341b8, 891e5d429fd64062b7791ddf6584ee3b; no voters:
I20260926 12:39:02.463356 19189 raft_consensus.cc:2781] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:39:02.463405 19189 raft_consensus.cc:482] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:39:02.463452 19189 raft_consensus.cc:3037] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.464761 19189 raft_consensus.cc:504] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.465150 19189 leader_election.cc:288] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [CANDIDATE]: Term 1 election: Requested vote from peers 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957), 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775)
I20260926 12:39:02.465548 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cce78f369047879b2b6efe212b5cd8" candidate_uuid: "891e5d429fd64062b7791ddf6584ee3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529c76a0c0824948947993f0cabee993"
I20260926 12:39:02.465672 19106 raft_consensus.cc:3037] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.466857 19106 raft_consensus.cc:2454] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 891e5d429fd64062b7791ddf6584ee3b in term 1.
I20260926 12:39:02.467113 18826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cce78f369047879b2b6efe212b5cd8" candidate_uuid: "891e5d429fd64062b7791ddf6584ee3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8"
I20260926 12:39:02.467252 18826 raft_consensus.cc:3037] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.467185 18900 leader_election.cc:302] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529c76a0c0824948947993f0cabee993, 891e5d429fd64062b7791ddf6584ee3b; no voters:
I20260926 12:39:02.467440 19189 raft_consensus.cc:2781] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:02.467733 19189 raft_consensus.cc:686] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [term 1 LEADER]: Becoming Leader. State: Replica: 891e5d429fd64062b7791ddf6584ee3b, State: Running, Role: LEADER
I20260926 12:39:02.468019 19189 consensus_queue.cc:237] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.468504 18826 raft_consensus.cc:2454] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 891e5d429fd64062b7791ddf6584ee3b in term 1.
I20260926 12:39:02.469559 19224 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: TxnStatusTablet state changed. Reason: New leader 891e5d429fd64062b7791ddf6584ee3b. Latest consensus state: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.469740 19224 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:39:02.470139 19226 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:39:02.470287 19226 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:39:02.470716 18667 catalog_manager.cc:5495] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130). New cstate: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } health_report { overall_health: UNKNOWN } } }
I20260926 12:39:02.488955 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.484049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41715 (local address 127.0.0.1:37172)
0926 12:39:02.484203 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.484219 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.484337 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.484718 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.484720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.484727 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:39:02.484823 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.484828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.486038 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.486041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.486708 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.486718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.487373 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.487376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.487453 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.487501 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.488637 (+ 1136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.488653 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.488724 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.488828 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 12:39:02.488986 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.484185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.10.224.130:41715)
0926 12:39:02.484311 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.484315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.484329 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:39:02.484483 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.484487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.484586 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.484706 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:39:02.484926 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.485884 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.486837 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.487218 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.487334 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.487716 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.488426 (+ 710us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.488585 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.488770 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.488833 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 12:39:02.492185 19106 raft_consensus.cc:1261] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Refusing update from remote peer 891e5d429fd64062b7791ddf6584ee3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:02.492374 18826 raft_consensus.cc:1261] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Refusing update from remote peer 891e5d429fd64062b7791ddf6584ee3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:02.492767 19189 consensus_queue.cc:1035] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:02.492996 19224 consensus_queue.cc:1035] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:02.494841 19187 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: TxnStatusTablet state changed. Reason: New leader 891e5d429fd64062b7791ddf6584ee3b. Latest consensus state: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.494985 19187 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:39:02.494894 19188 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: TxnStatusTablet state changed. Reason: New leader 891e5d429fd64062b7791ddf6584ee3b. Latest consensus state: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.495030 19188 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:39:02.497400 19224 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.497535 19224 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:39:02.497530 19194 mvcc.cc:205] Tried to move back new op lower bound from 7333586298842005504 to 7333586298752540672. Current Snapshot: MvccSnapshot[applied={T|T < 7333586298842005504}]
I20260926 12:39:02.497871 19224 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.497960 19224 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 891e5d429fd64062b7791ddf6584ee3b: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:39:02.498057 19195 mvcc.cc:205] Tried to move back new op lower bound from 7333586298842005504 to 7333586298752540672. Current Snapshot: MvccSnapshot[applied={T|T < 7333586298842005504 or (T in {7333586298842005504})}]
I20260926 12:39:02.498258 19188 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.498381 19188 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 1e6a5856e3b14ee9a95bf003bb4341b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:39:02.500032 19187 tablet_replica.cc:439] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } } }
I20260926 12:39:02.500160 19187 tablet_replica.cc:442] T 35cce78f369047879b2b6efe212b5cd8 P 529c76a0c0824948947993f0cabee993: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:39:02.509321 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.505093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.10.224.131:39775)
0926 12:39:02.505215 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.505219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.505233 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:39:02.505392 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.505396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.505468 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.505592 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:39:02.505795 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.506677 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.507571 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.507963 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.508083 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.508296 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.508921 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.509057 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.509103 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.509166 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:39:02.522404 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.518027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:44921 (local address 127.10.224.130:41715)
0926 12:39:02.518154 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.518157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.518472 (+ 315us) server_negotiation.cc:407] Connection header received
0926 12:39:02.518644 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.518647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.518703 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.518780 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:39:02.519098 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.519763 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.521136 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.521360 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.521443 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.521844 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.521926 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.522000 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.522209 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.522280 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:39:02.522397 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.517927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41715 (local address 127.10.224.131:44921)
0926 12:39:02.518371 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.518387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.518513 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.518807 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.518810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.518819 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:39:02.518946 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.518955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.519871 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.519875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.521006 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.521018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.521522 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.521527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.521627 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.521692 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.522016 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.522023 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.522105 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.522199 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":140,"threads_started":1}
I20260926 12:39:02.529808 18826 raft_consensus.cc:1261] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Refusing update from remote peer 529c76a0c0824948947993f0cabee993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:02.529889 18966 raft_consensus.cc:1261] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Refusing update from remote peer 529c76a0c0824948947993f0cabee993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:02.530310 19211 consensus_queue.cc:1035] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:02.530603 19187 consensus_queue.cc:1035] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:02.563987 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.551828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:39775 (local address 127.10.224.130:48483)
0926 12:39:02.559767 (+ 7939us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:02.559783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:02.559899 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:02.560217 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:02.560221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:02.560266 (+ 45us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:39:02.560393 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.560402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.561486 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.561490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.562297 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:02.562307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.563047 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.563050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:02.563140 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.563204 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.563646 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.563653 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:39:02.563730 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:39:02.563811 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7796,"thread_start_us":131,"threads_started":1}
I20260926 12:39:02.563994 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.551905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:48483 (local address 127.10.224.131:39775)
0926 12:39:02.552034 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.552043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.559882 (+ 7839us) server_negotiation.cc:407] Connection header received
0926 12:39:02.560042 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.560045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.560110 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.560223 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:39:02.560600 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.561350 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.562491 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.562880 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.563014 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.563396 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.563497 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.563618 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.563761 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.563830 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:39:02.598515 18666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:02.598707 18666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:02.601413 18666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: Sending DeleteTablet for 3 replicas of tablet ecee785858eb4b89b93b10722844c569
I20260926 12:39:02.602448 18806 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee785858eb4b89b93b10722844c569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:40562
I20260926 12:39:02.602427 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee785858eb4b89b93b10722844c569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:51568
I20260926 12:39:02.602665 19206 tablet_replica.cc:330] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: stopping tablet replica
I20260926 12:39:02.602667 19205 tablet_replica.cc:330] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: stopping tablet replica
I20260926 12:39:02.602797 19205 raft_consensus.cc:2229] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:02.602797 19206 raft_consensus.cc:2229] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:39:02.602926 19205 raft_consensus.cc:2258] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.603157 19206 raft_consensus.cc:2258] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.603248 18946 tablet_service.cc:1514] Processing DeleteTablet for tablet ecee785858eb4b89b93b10722844c569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:37170
I20260926 12:39:02.603427 19204 tablet_replica.cc:330] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: stopping tablet replica
I20260926 12:39:02.603561 19204 raft_consensus.cc:2229] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:02.603674 19204 raft_consensus.cc:2258] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.604005 19205 ts_tablet_manager.cc:1905] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.604506 19204 ts_tablet_manager.cc:1905] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.607133 19205 ts_tablet_manager.cc:1918] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:39:02.607209 19205 log.cc:1196] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/ecee785858eb4b89b93b10722844c569
I20260926 12:39:02.607236 19204 ts_tablet_manager.cc:1918] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:39:02.607303 19204 log.cc:1196] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/ecee785858eb4b89b93b10722844c569
I20260926 12:39:02.607488 19205 ts_tablet_manager.cc:1939] T ecee785858eb4b89b93b10722844c569 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting consensus metadata
I20260926 12:39:02.607589 19204 ts_tablet_manager.cc:1939] T ecee785858eb4b89b93b10722844c569 P 891e5d429fd64062b7791ddf6584ee3b: Deleting consensus metadata
I20260926 12:39:02.608335 18656 catalog_manager.cc:4841] TS 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715): tablet ecee785858eb4b89b93b10722844c569 (table test-table [id=9a91a0d2be464f8e816bef44b32c826c]) successfully deleted
I20260926 12:39:02.608367 19206 ts_tablet_manager.cc:1905] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.611593 19206 ts_tablet_manager.cc:1918] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:39:02.611696 19206 log.cc:1196] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/ecee785858eb4b89b93b10722844c569
I20260926 12:39:02.612874 18655 catalog_manager.cc:4841] TS 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957): tablet ecee785858eb4b89b93b10722844c569 (table test-table [id=9a91a0d2be464f8e816bef44b32c826c]) successfully deleted
I20260926 12:39:02.616659 19206 ts_tablet_manager.cc:1939] T ecee785858eb4b89b93b10722844c569 P 529c76a0c0824948947993f0cabee993: Deleting consensus metadata
I20260926 12:39:02.617558 18656 catalog_manager.cc:4841] TS 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775): tablet ecee785858eb4b89b93b10722844c569 (table test-table [id=9a91a0d2be464f8e816bef44b32c826c]) successfully deleted
Sep 26 12:39:02 dist-test-slave-m3pj krb5kdc[18624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:39:02 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:02 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:39:02.648600 18718 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:02.640080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.10.224.190:36275)
0926 12:39:02.640240 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.640245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.640703 (+ 458us) server_negotiation.cc:407] Connection header received
0926 12:39:02.640840 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.640845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.640907 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.641020 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:02.641901 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.642777 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.644046 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.644429 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.644560 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.648424 (+ 3864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39748: BlockingRecv error: recv got EOF from 127.0.0.1:39748 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 12:39:02 dist-test-slave-m3pj krb5kdc[18624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM
I20260926 12:39:02.663049 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.650966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.10.224.190:36275)
0926 12:39:02.651128 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.651133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.651606 (+ 473us) server_negotiation.cc:407] Connection header received
0926 12:39:02.651761 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.651765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.651846 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.651941 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:02.652787 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.653799 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.655045 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.655437 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.655618 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.658813 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:02.658818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:02.658821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:02.658859 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:02.661014 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:02.661592 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:02.661596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:02.661598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:02.661705 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:02.662035 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:02.662039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:02.662041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:02.662314 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:02.662468 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.662718 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.662855 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 12:39:02.666146 18666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:39:02.666666 18666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:39:02.672703 18806 tablet_service.cc:1467] Processing CreateTablet for tablet f13dbda61e434f5bbabe0234c3d0839a (DEFAULT_TABLE table=test-table [id=a4079598dccf4be0bed15c78903578c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.672704 19086 tablet_service.cc:1467] Processing CreateTablet for tablet f13dbda61e434f5bbabe0234c3d0839a (DEFAULT_TABLE table=test-table [id=a4079598dccf4be0bed15c78903578c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.673035 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13dbda61e434f5bbabe0234c3d0839a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.673040 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13dbda61e434f5bbabe0234c3d0839a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.675709 19181 tablet_bootstrap.cc:492] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: Bootstrap starting.
I20260926 12:39:02.675813 18946 tablet_service.cc:1467] Processing CreateTablet for tablet f13dbda61e434f5bbabe0234c3d0839a (DEFAULT_TABLE table=test-table [id=a4079598dccf4be0bed15c78903578c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.676123 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13dbda61e434f5bbabe0234c3d0839a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.676868 19181 tablet_bootstrap.cc:654] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.678680 19182 tablet_bootstrap.cc:492] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: Bootstrap starting.
I20260926 12:39:02.679605 19182 tablet_bootstrap.cc:654] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.680518 19180 tablet_bootstrap.cc:492] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: Bootstrap starting.
I20260926 12:39:02.680914 19181 tablet_bootstrap.cc:492] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: No bootstrap required, opened a new log
I20260926 12:39:02.681025 19181 ts_tablet_manager.cc:1397] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 12:39:02.681645 19180 tablet_bootstrap.cc:654] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.681679 19181 raft_consensus.cc:348] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.681820 19181 raft_consensus.cc:374] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.681901 19181 raft_consensus.cc:729] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529c76a0c0824948947993f0cabee993, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.682072 19181 consensus_queue.cc:260] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.682412 19181 ts_tablet_manager.cc:1428] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 12:39:02.685403 19182 tablet_bootstrap.cc:492] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: No bootstrap required, opened a new log
I20260926 12:39:02.685556 19182 ts_tablet_manager.cc:1397] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260926 12:39:02.685997 19182 raft_consensus.cc:348] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.686123 19182 raft_consensus.cc:374] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.686178 19182 raft_consensus.cc:729] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891e5d429fd64062b7791ddf6584ee3b, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.686307 19182 consensus_queue.cc:260] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.686592 19182 ts_tablet_manager.cc:1428] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:39:02.688474 19180 tablet_bootstrap.cc:492] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: No bootstrap required, opened a new log
I20260926 12:39:02.688580 19180 ts_tablet_manager.cc:1397] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260926 12:39:02.689003 19180 raft_consensus.cc:348] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.689122 19180 raft_consensus.cc:374] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.689153 19180 raft_consensus.cc:729] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6a5856e3b14ee9a95bf003bb4341b8, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.689268 19180 consensus_queue.cc:260] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.689698 19180 ts_tablet_manager.cc:1428] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 12:39:02.850703 19236 raft_consensus.cc:482] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:39:02.850859 19236 raft_consensus.cc:504] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.851404 19236 leader_election.cc:288] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957), 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775)
I20260926 12:39:02.851862 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13dbda61e434f5bbabe0234c3d0839a" candidate_uuid: "891e5d429fd64062b7791ddf6584ee3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529c76a0c0824948947993f0cabee993" is_pre_election: true
I20260926 12:39:02.851856 18825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13dbda61e434f5bbabe0234c3d0839a" candidate_uuid: "891e5d429fd64062b7791ddf6584ee3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" is_pre_election: true
I20260926 12:39:02.852025 19106 raft_consensus.cc:2454] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 891e5d429fd64062b7791ddf6584ee3b in term 0.
I20260926 12:39:02.852047 18825 raft_consensus.cc:2454] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 891e5d429fd64062b7791ddf6584ee3b in term 0.
I20260926 12:39:02.852366 18900 leader_election.cc:302] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529c76a0c0824948947993f0cabee993, 891e5d429fd64062b7791ddf6584ee3b; no voters:
I20260926 12:39:02.852564 19236 raft_consensus.cc:2781] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:39:02.852632 19236 raft_consensus.cc:482] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:39:02.852672 19236 raft_consensus.cc:3037] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.853786 19236 raft_consensus.cc:504] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.854168 19236 leader_election.cc:288] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [CANDIDATE]: Term 1 election: Requested vote from peers 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957), 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775)
I20260926 12:39:02.854573 18825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13dbda61e434f5bbabe0234c3d0839a" candidate_uuid: "891e5d429fd64062b7791ddf6584ee3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8"
I20260926 12:39:02.854573 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13dbda61e434f5bbabe0234c3d0839a" candidate_uuid: "891e5d429fd64062b7791ddf6584ee3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529c76a0c0824948947993f0cabee993"
I20260926 12:39:02.854710 18825 raft_consensus.cc:3037] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.854722 19106 raft_consensus.cc:3037] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:02.855777 19106 raft_consensus.cc:2454] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 891e5d429fd64062b7791ddf6584ee3b in term 1.
I20260926 12:39:02.855789 18825 raft_consensus.cc:2454] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 891e5d429fd64062b7791ddf6584ee3b in term 1.
I20260926 12:39:02.856113 18900 leader_election.cc:302] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529c76a0c0824948947993f0cabee993, 891e5d429fd64062b7791ddf6584ee3b; no voters:
I20260926 12:39:02.856312 19236 raft_consensus.cc:2781] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:02.856406 19236 raft_consensus.cc:686] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 1 LEADER]: Becoming Leader. State: Replica: 891e5d429fd64062b7791ddf6584ee3b, State: Running, Role: LEADER
I20260926 12:39:02.856547 19236 consensus_queue.cc:237] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.858172 18667 catalog_manager.cc:5495] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130). New cstate: current_term: 1 leader_uuid: "891e5d429fd64062b7791ddf6584ee3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } health_report { overall_health: UNKNOWN } } }
I20260926 12:39:02.883502 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:02.879702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.10.224.130:41715)
0926 12:39:02.879863 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:02.879870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:02.879884 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:39:02.879941 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:02.879944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:02.880001 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:02.880083 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:39:02.880336 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.880843 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.882240 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:02.882550 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:02.882659 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:02.882962 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:39:02.883049 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:39:02.883129 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:02.883248 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 12:39:02.883309 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 12:39:02.886389 18825 raft_consensus.cc:1261] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Refusing update from remote peer 891e5d429fd64062b7791ddf6584ee3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:02.886467 19106 raft_consensus.cc:1261] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Refusing update from remote peer 891e5d429fd64062b7791ddf6584ee3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:02.886909 19189 consensus_queue.cc:1035] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:02.887082 19236 consensus_queue.cc:1035] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:02.897974 18667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:02.898156 18667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:02.900553 18667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: Sending DeleteTablet for 3 replicas of tablet f13dbda61e434f5bbabe0234c3d0839a
I20260926 12:39:02.901306 18806 tablet_service.cc:1514] Processing DeleteTablet for tablet f13dbda61e434f5bbabe0234c3d0839a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:40562
I20260926 12:39:02.901474 18946 tablet_service.cc:1514] Processing DeleteTablet for tablet f13dbda61e434f5bbabe0234c3d0839a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:37170
I20260926 12:39:02.901535 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet f13dbda61e434f5bbabe0234c3d0839a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:02 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:51568
I20260926 12:39:02.901713 19206 tablet_replica.cc:330] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: stopping tablet replica
I20260926 12:39:02.901716 19205 tablet_replica.cc:330] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: stopping tablet replica
I20260926 12:39:02.901844 19206 raft_consensus.cc:2229] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:02.901865 19205 raft_consensus.cc:2229] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:02.902009 19206 raft_consensus.cc:2258] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.902052 19205 raft_consensus.cc:2258] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.902256 19204 tablet_replica.cc:330] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: stopping tablet replica
I20260926 12:39:02.902395 19204 raft_consensus.cc:2229] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:39:02.902743 19204 raft_consensus.cc:2258] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:02.902758 19206 ts_tablet_manager.cc:1905] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.903482 19204 ts_tablet_manager.cc:1905] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.904266 18667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:39:02.904752 18667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:39:02.905393 19206 ts_tablet_manager.cc:1918] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:39:02.905459 19206 log.cc:1196] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/f13dbda61e434f5bbabe0234c3d0839a
I20260926 12:39:02.905717 19206 ts_tablet_manager.cc:1939] T f13dbda61e434f5bbabe0234c3d0839a P 529c76a0c0824948947993f0cabee993: Deleting consensus metadata
I20260926 12:39:02.906318 19204 ts_tablet_manager.cc:1918] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:39:02.906395 19204 log.cc:1196] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/f13dbda61e434f5bbabe0234c3d0839a
I20260926 12:39:02.906546 18656 catalog_manager.cc:4841] TS 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775): tablet f13dbda61e434f5bbabe0234c3d0839a (table test-table [id=a4079598dccf4be0bed15c78903578c5]) successfully deleted
I20260926 12:39:02.906688 19204 ts_tablet_manager.cc:1939] T f13dbda61e434f5bbabe0234c3d0839a P 891e5d429fd64062b7791ddf6584ee3b: Deleting consensus metadata
I20260926 12:39:02.907253 19205 ts_tablet_manager.cc:1905] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:02.909386 19205 ts_tablet_manager.cc:1918] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:39:02.909504 19205 log.cc:1196] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/f13dbda61e434f5bbabe0234c3d0839a
I20260926 12:39:02.909760 19205 ts_tablet_manager.cc:1939] T f13dbda61e434f5bbabe0234c3d0839a P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting consensus metadata
I20260926 12:39:02.910442 18806 tablet_service.cc:1467] Processing CreateTablet for tablet 502773c23cd2423c88aab2dc78fc0418 (DEFAULT_TABLE table=test-table [id=f73c7bde38cb4ec091578106dbdf2648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.910701 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502773c23cd2423c88aab2dc78fc0418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.911751 18655 catalog_manager.cc:4841] TS 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957): tablet f13dbda61e434f5bbabe0234c3d0839a (table test-table [id=a4079598dccf4be0bed15c78903578c5]) successfully deleted
I20260926 12:39:02.911959 18656 catalog_manager.cc:4841] TS 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715): tablet f13dbda61e434f5bbabe0234c3d0839a (table test-table [id=a4079598dccf4be0bed15c78903578c5]) successfully deleted
I20260926 12:39:02.912485 18946 tablet_service.cc:1467] Processing CreateTablet for tablet 502773c23cd2423c88aab2dc78fc0418 (DEFAULT_TABLE table=test-table [id=f73c7bde38cb4ec091578106dbdf2648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.912801 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502773c23cd2423c88aab2dc78fc0418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.912917 19180 tablet_bootstrap.cc:492] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: Bootstrap starting.
I20260926 12:39:02.913242 19086 tablet_service.cc:1467] Processing CreateTablet for tablet 502773c23cd2423c88aab2dc78fc0418 (DEFAULT_TABLE table=test-table [id=f73c7bde38cb4ec091578106dbdf2648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:02.913479 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502773c23cd2423c88aab2dc78fc0418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:02.913843 19180 tablet_bootstrap.cc:654] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.915266 19182 tablet_bootstrap.cc:492] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: Bootstrap starting.
I20260926 12:39:02.916145 19181 tablet_bootstrap.cc:492] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: Bootstrap starting.
I20260926 12:39:02.916290 19182 tablet_bootstrap.cc:654] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.917003 19181 tablet_bootstrap.cc:654] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:02.917933 19180 tablet_bootstrap.cc:492] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: No bootstrap required, opened a new log
I20260926 12:39:02.918052 19180 ts_tablet_manager.cc:1397] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 12:39:02.918124 19181 tablet_bootstrap.cc:492] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: No bootstrap required, opened a new log
I20260926 12:39:02.918224 19181 ts_tablet_manager.cc:1397] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:39:02.918516 19180 raft_consensus.cc:348] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.918658 19180 raft_consensus.cc:374] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.918618 19181 raft_consensus.cc:348] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.918704 19180 raft_consensus.cc:729] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6a5856e3b14ee9a95bf003bb4341b8, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.918727 19181 raft_consensus.cc:374] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.918782 19181 raft_consensus.cc:729] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529c76a0c0824948947993f0cabee993, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.918856 19180 consensus_queue.cc:260] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.918897 19181 consensus_queue.cc:260] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.919158 19181 ts_tablet_manager.cc:1428] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:39:02.919724 19182 tablet_bootstrap.cc:492] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: No bootstrap required, opened a new log
I20260926 12:39:02.919834 19182 ts_tablet_manager.cc:1397] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260926 12:39:02.920225 19182 raft_consensus.cc:348] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.920351 19182 raft_consensus.cc:374] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:02.920418 19182 raft_consensus.cc:729] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891e5d429fd64062b7791ddf6584ee3b, State: Initialized, Role: FOLLOWER
I20260926 12:39:02.920569 19182 consensus_queue.cc:260] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:02.920893 19182 ts_tablet_manager.cc:1428] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:39:02.921944 19180 ts_tablet_manager.cc:1428] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 12:39:03.004566 19211 raft_consensus.cc:482] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:39:03.004736 19211 raft_consensus.cc:504] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:03.005223 19211 leader_election.cc:288] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957), 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715)
I20260926 12:39:03.005666 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502773c23cd2423c88aab2dc78fc0418" candidate_uuid: "529c76a0c0824948947993f0cabee993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891e5d429fd64062b7791ddf6584ee3b" is_pre_election: true
I20260926 12:39:03.005686 18825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502773c23cd2423c88aab2dc78fc0418" candidate_uuid: "529c76a0c0824948947993f0cabee993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" is_pre_election: true
I20260926 12:39:03.005861 18966 raft_consensus.cc:2454] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529c76a0c0824948947993f0cabee993 in term 0.
I20260926 12:39:03.005877 18825 raft_consensus.cc:2454] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529c76a0c0824948947993f0cabee993 in term 0.
I20260926 12:39:03.006161 19040 leader_election.cc:302] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529c76a0c0824948947993f0cabee993, 891e5d429fd64062b7791ddf6584ee3b; no voters:
I20260926 12:39:03.006364 19211 raft_consensus.cc:2781] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:39:03.006443 19211 raft_consensus.cc:482] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:39:03.006496 19211 raft_consensus.cc:3037] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:03.007618 19211 raft_consensus.cc:504] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:03.008029 19211 leader_election.cc:288] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [CANDIDATE]: Term 1 election: Requested vote from peers 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957), 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715)
I20260926 12:39:03.008433 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502773c23cd2423c88aab2dc78fc0418" candidate_uuid: "529c76a0c0824948947993f0cabee993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891e5d429fd64062b7791ddf6584ee3b"
I20260926 12:39:03.008469 18825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502773c23cd2423c88aab2dc78fc0418" candidate_uuid: "529c76a0c0824948947993f0cabee993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8"
I20260926 12:39:03.008617 18825 raft_consensus.cc:3037] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:03.008620 18966 raft_consensus.cc:3037] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:03.009588 18825 raft_consensus.cc:2454] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529c76a0c0824948947993f0cabee993 in term 1.
I20260926 12:39:03.009933 19039 leader_election.cc:302] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6a5856e3b14ee9a95bf003bb4341b8, 529c76a0c0824948947993f0cabee993; no voters:
I20260926 12:39:03.009970 18966 raft_consensus.cc:2454] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529c76a0c0824948947993f0cabee993 in term 1.
I20260926 12:39:03.010136 19211 raft_consensus.cc:2781] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:03.010233 19211 raft_consensus.cc:686] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 1 LEADER]: Becoming Leader. State: Replica: 529c76a0c0824948947993f0cabee993, State: Running, Role: LEADER
I20260926 12:39:03.010389 19211 consensus_queue.cc:237] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } }
I20260926 12:39:03.012372 18667 catalog_manager.cc:5495] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 reported cstate change: term changed from 0 to 1, leader changed from <none> to 529c76a0c0824948947993f0cabee993 (127.10.224.131). New cstate: current_term: 1 leader_uuid: "529c76a0c0824948947993f0cabee993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529c76a0c0824948947993f0cabee993" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 39775 } health_report { overall_health: HEALTHY } } }
I20260926 12:39:03.038151 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:03.034766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.10.224.131:39775)
0926 12:39:03.034879 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:03.034883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:03.034899 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:39:03.035021 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:03.035023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:03.035072 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:03.035159 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:39:03.035386 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:03.036194 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:03.037042 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:03.037377 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:03.037480 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:03.037697 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:39:03.037782 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:39:03.037849 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:03.037934 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 12:39:03.037998 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:39:03.045327 18825 raft_consensus.cc:1261] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Refusing update from remote peer 529c76a0c0824948947993f0cabee993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:03.045411 18966 raft_consensus.cc:1261] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Refusing update from remote peer 529c76a0c0824948947993f0cabee993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:03.045923 19211 consensus_queue.cc:1035] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e6a5856e3b14ee9a95bf003bb4341b8" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:03.046168 19187 consensus_queue.cc:1035] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "891e5d429fd64062b7791ddf6584ee3b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:03.050918 19196 mvcc.cc:205] Tried to move back new op lower bound from 7333586301110108160 to 7333586300972310528. Current Snapshot: MvccSnapshot[applied={T|T < 7333586301110108160}]
I20260926 12:39:03.084738 18667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:03.084914 18667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:03.087416 18667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd00e4e691234b54b8742a40dfb0e62e: Sending DeleteTablet for 3 replicas of tablet 502773c23cd2423c88aab2dc78fc0418
I20260926 12:39:03.088169 18806 tablet_service.cc:1514] Processing DeleteTablet for tablet 502773c23cd2423c88aab2dc78fc0418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:03 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:40562
I20260926 12:39:03.088204 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet 502773c23cd2423c88aab2dc78fc0418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:03 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:51568
I20260926 12:39:03.088204 18946 tablet_service.cc:1514] Processing DeleteTablet for tablet 502773c23cd2423c88aab2dc78fc0418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:03 UTC) from {username='oryx', principal='oryx/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:37170
I20260926 12:39:03.088418 19206 tablet_replica.cc:330] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: stopping tablet replica
I20260926 12:39:03.088450 19205 tablet_replica.cc:330] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: stopping tablet replica
I20260926 12:39:03.088572 19205 raft_consensus.cc:2229] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:03.088565 19206 raft_consensus.cc:2229] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:39:03.088711 19205 raft_consensus.cc:2258] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:03.088722 19204 tablet_replica.cc:330] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: stopping tablet replica
I20260926 12:39:03.088831 19206 raft_consensus.cc:2258] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:03.088865 19204 raft_consensus.cc:2229] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:03.089001 19204 raft_consensus.cc:2258] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:03.089645 19205 ts_tablet_manager.cc:1905] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:03.089645 19206 ts_tablet_manager.cc:1905] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:03.089932 19204 ts_tablet_manager.cc:1905] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:03.092517 19205 ts_tablet_manager.cc:1918] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:39:03.092517 19206 ts_tablet_manager.cc:1918] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:39:03.092610 19206 log.cc:1196] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/502773c23cd2423c88aab2dc78fc0418
I20260926 12:39:03.092610 19205 log.cc:1196] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/502773c23cd2423c88aab2dc78fc0418
I20260926 12:39:03.092877 19206 ts_tablet_manager.cc:1939] T 502773c23cd2423c88aab2dc78fc0418 P 529c76a0c0824948947993f0cabee993: Deleting consensus metadata
I20260926 12:39:03.092878 19205 ts_tablet_manager.cc:1939] T 502773c23cd2423c88aab2dc78fc0418 P 1e6a5856e3b14ee9a95bf003bb4341b8: Deleting consensus metadata
I20260926 12:39:03.093088 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 18741
I20260926 12:39:03.094565 19204 ts_tablet_manager.cc:1918] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:39:03.094666 19204 log.cc:1196] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/502773c23cd2423c88aab2dc78fc0418
I20260926 12:39:03.094944 19204 ts_tablet_manager.cc:1939] T 502773c23cd2423c88aab2dc78fc0418 P 891e5d429fd64062b7791ddf6584ee3b: Deleting consensus metadata
I20260926 12:39:03.095304 18656 catalog_manager.cc:4841] TS 529c76a0c0824948947993f0cabee993 (127.10.224.131:39775): tablet 502773c23cd2423c88aab2dc78fc0418 (table test-table [id=f73c7bde38cb4ec091578106dbdf2648]) successfully deleted
I20260926 12:39:03.095744 18656 catalog_manager.cc:4841] TS 891e5d429fd64062b7791ddf6584ee3b (127.10.224.130:41715): tablet 502773c23cd2423c88aab2dc78fc0418 (table test-table [id=f73c7bde38cb4ec091578106dbdf2648]) successfully deleted
I20260926 12:39:03.100778 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 18881
W20260926 12:39:03.100778 18655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:40957 (error 108)
I20260926 12:39:03.102672 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:03.102130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40957 (local address 127.0.0.1:40576)
0926 12:39:03.102419 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:03.102530 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:40957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
W20260926 12:39:03.102834 18655 catalog_manager.cc:4584] TS 1e6a5856e3b14ee9a95bf003bb4341b8 (127.10.224.129:40957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 502773c23cd2423c88aab2dc78fc0418: Network error: Client connection negotiation failed: client connection to 127.10.224.129:40957: connect: Connection refused (error 111)
I20260926 12:39:03.108857 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 19021
I20260926 12:39:03.115998 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 18637
2026-09-26T12:39:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3389 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:39:03 dist-test-slave-m3pj krb5kdc[19274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:39:03 dist-test-slave-m3pj krb5kdc[19274](info): set up 2 sockets
Sep 26 12:39:03 dist-test-slave-m3pj krb5kdc[19274](info): commencing operation
krb5kdc: starting...
W20260926 12:39:04.192801 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:39:04 dist-test-slave-m3pj krb5kdc[19274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:39:04.328086 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38211
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:04.448014 19287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:04.448357 19287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:04.448403 19287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:04.452795 19287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:39:04.452878 19287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:04.452899 19287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:04.452917 19287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:39:04.452932 19287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:39:04.458926 19287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:38211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:04.460247 19287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:04.461447 19287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:04.471359 19295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:04.471689 19287 server_base.cc:1034] running on GCE node
W20260926 12:39:04.472682 19293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:04.473207 19292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:04.474066 19287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:04.475391 19287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:04.476622 19287 hybrid_clock.cc:648] HybridClock initialized: now 1790426344476603 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T12:39:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1379 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:39:04 dist-test-slave-m3pj krb5kdc[19300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:39:04 dist-test-slave-m3pj krb5kdc[19300](info): set up 2 sockets
Sep 26 12:39:04 dist-test-slave-m3pj krb5kdc[19300](info): commencing operation
krb5kdc: starting...
W20260926 12:39:05.632891 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:39:05 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:39:05.767733 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:44809
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:05.883283 19313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:05.883683 19313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:05.883771 19313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:05.888103 19313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:39:05.888229 19313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:05.888265 19313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:05.888304 19313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:39:05.888341 19313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:39:05.894234 19313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:05.895466 19313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:05.896618 19313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:05.906662 19318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:05.906777 19319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:05.906662 19321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:05.906761 19313 server_base.cc:1034] running on GCE node
I20260926 12:39:05.907644 19313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:05.908866 19313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:05.910081 19313 hybrid_clock.cc:648] HybridClock initialized: now 1790426345910063 us; error 38 us; skew 500 ppm
Sep 26 12:39:05 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426345, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:05.913314 19313 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:39:05.914521 19313 webserver.cc:460] Webserver started at http://127.10.224.190:43911/ using document root <none> and password file <none>
I20260926 12:39:05.915175 19313 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:05.915282 19313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:05.915642 19313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:05.918140 19313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "581c6c9b2b1f4b1f9726d929da1557ef"
format_stamp: "Formatted at 2026-09-26 12:39:05 on dist-test-slave-m3pj"
server_key: "f86752f11228ee8f4bc2d32769cf7c39"
server_key_iv: "7a9f694936969c5bc6f6785a9f25045f"
server_key_version: "encryptionkey@0"
I20260926 12:39:05.918913 19313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "581c6c9b2b1f4b1f9726d929da1557ef"
format_stamp: "Formatted at 2026-09-26 12:39:05 on dist-test-slave-m3pj"
server_key: "f86752f11228ee8f4bc2d32769cf7c39"
server_key_iv: "7a9f694936969c5bc6f6785a9f25045f"
server_key_version: "encryptionkey@0"
I20260926 12:39:05.923640 19313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 12:39:05.926810 19328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:05.928519 19313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:05.928712 19313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "581c6c9b2b1f4b1f9726d929da1557ef"
format_stamp: "Formatted at 2026-09-26 12:39:05 on dist-test-slave-m3pj"
server_key: "f86752f11228ee8f4bc2d32769cf7c39"
server_key_iv: "7a9f694936969c5bc6f6785a9f25045f"
server_key_version: "encryptionkey@0"
I20260926 12:39:05.928864 19313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:05.946519 19313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:05.950165 19313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:05.950416 19313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:05.958557 19313 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:44809
I20260926 12:39:05.958663 19380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:44809 every 8 connection(s)
I20260926 12:39:05.959769 19313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:39:05.962827 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:05.965505 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 19313
I20260926 12:39:05.965651 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:39:05.965941 11138 external_mini_cluster.cc:1428] Setting key d24d78db3802c4a561e8f90d43e55613
I20260926 12:39:05.969985 19381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Bootstrap starting.
I20260926 12:39:05.972786 19381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:05.973711 19381 log.cc:824] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:05.976151 19381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: No bootstrap required, opened a new log
Sep 26 12:39:05 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:39:05.980399 19381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } }
I20260926 12:39:05.980655 19381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:05.980691 19381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581c6c9b2b1f4b1f9726d929da1557ef, State: Initialized, Role: FOLLOWER
I20260926 12:39:05.981235 19381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } }
I20260926 12:39:05.981417 19381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:39:05.981505 19381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:39:05.981650 19381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:05.982568 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:05.967884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.10.224.190:44809)
0926 12:39:05.968450 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:05.968475 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:05.968539 (+ 64us) server_negotiation.cc:407] Connection header received
0926 12:39:05.969277 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:05.969300 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:05.969695 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:05.970042 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:05.971005 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:05.972171 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:05.973352 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:05.973923 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:05.974055 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:05.976984 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:05.976990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:05.977005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:05.977036 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:05.979042 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:05.979659 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:05.979666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:05.979672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:05.979852 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:05.980212 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:05.980215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:05.980217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:05.980774 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:05.981044 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:05.981327 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 12:39:05.981684 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":178,"threads_started":1}
I20260926 12:39:05.982695 19381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } }
I20260926 12:39:05.983098 19381 leader_election.cc:302] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 581c6c9b2b1f4b1f9726d929da1557ef; no voters:
I20260926 12:39:05.983480 19381 leader_election.cc:288] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:39:05.983629 19386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:05.983870 19386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 1 LEADER]: Becoming Leader. State: Replica: 581c6c9b2b1f4b1f9726d929da1557ef, State: Running, Role: LEADER
I20260926 12:39:05.984264 19386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } }
I20260926 12:39:05.984894 19381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:39:05.985584 19388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 581c6c9b2b1f4b1f9726d929da1557ef. Latest consensus state: current_term: 1 leader_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } } }
I20260926 12:39:05.985728 19388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:05.986003 19387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } } }
I20260926 12:39:05.986099 19387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:05.986159 19392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:39:05.989722 19392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:39:05.997011 19392 catalog_manager.cc:1347] Generated new cluster ID: a9e8359cd7ba4f328af147c9a66099db
I20260926 12:39:05.997117 19392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:39:06.014421 19392 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:39:06.015901 19392 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:39:06.021122 19392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Generated new TSK 0
I20260926 12:39:06.022126 19392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:39:06.059696 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44809
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:06.180434 19407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:06.180732 19407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:06.180778 19407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:06.184950 19407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:06.185021 19407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:06.185086 19407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:39:06.191007 19407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:06.192129 19407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:06.193125 19407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:06.203444 19412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:06.203570 19407 server_base.cc:1034] running on GCE node
W20260926 12:39:06.203438 19413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:06.203444 19415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:06.204162 19407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:06.204875 19407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:06.206070 19407 hybrid_clock.cc:648] HybridClock initialized: now 1790426346206029 us; error 56 us; skew 500 ppm
Sep 26 12:39:06 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:06.209512 19407 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:39:06.210471 19407 webserver.cc:460] Webserver started at http://127.10.224.129:42913/ using document root <none> and password file <none>
I20260926 12:39:06.211074 19407 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:06.211174 19407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:06.211434 19407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:06.213625 19407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "482cf573ff544526b11458828f56efe7"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "9304087e2d03aac586a7a4587c280a56"
server_key_iv: "47d47649716dd2b9262f5d369fd8bfc1"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.214205 19407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "482cf573ff544526b11458828f56efe7"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "9304087e2d03aac586a7a4587c280a56"
server_key_iv: "47d47649716dd2b9262f5d369fd8bfc1"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.218556 19407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:39:06.221606 19422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.223229 19407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:06.223460 19407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "482cf573ff544526b11458828f56efe7"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "9304087e2d03aac586a7a4587c280a56"
server_key_iv: "47d47649716dd2b9262f5d369fd8bfc1"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.223692 19407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:06.250286 19407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:06.254200 19407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:06.254456 19407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:06.255151 19407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:06.256145 19407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:06.256227 19407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.256322 19407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:06.256381 19407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.267123 19407 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:45647
I20260926 12:39:06.267155 19535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:45647 every 8 connection(s)
I20260926 12:39:06.268334 19407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:39:06.277210 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 19407
I20260926 12:39:06.277396 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:39:06.277745 11138 external_mini_cluster.cc:1428] Setting key b92e2254072980efac8d8e725602207c
Sep 26 12:39:06 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:39:06.285720 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:06.270629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:33299 (local address 127.10.224.190:44809)
0926 12:39:06.270836 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:06.270840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:06.272173 (+ 1333us) server_negotiation.cc:407] Connection header received
0926 12:39:06.273058 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:06.273061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:06.273113 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:06.273198 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:06.274515 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.275190 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.276668 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.276977 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.277133 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:06.280697 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:06.280701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:06.280703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:06.280732 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:06.282781 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:06.283734 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:06.283737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:06.283738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:06.283837 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:06.284346 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:06.284349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:06.284350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:06.284580 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:06.284712 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:06.285359 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 12:39:06.285543 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 12:39:06.286823 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:06.271085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:33299)
0926 12:39:06.272003 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:06.272052 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:06.272866 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:06.273378 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:06.273386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:06.273778 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:06.274335 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:06.274349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.275352 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.275355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:06.276514 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:06.276526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.277196 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.277201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:06.277501 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:06.278316 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:06.278343 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:06.280477 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:06.282968 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:06.282979 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:06.283002 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:06.283570 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:06.283944 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:06.283950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:06.283955 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:06.284192 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:06.284712 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:06.284721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:06.285098 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 12:39:06.285306 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 12:39:06.285745 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":135,"threads_started":1}
I20260926 12:39:06.288309 19536 heartbeater.cc:347] Connected to a master server at 127.10.224.190:44809
I20260926 12:39:06.288661 19536 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:39:06.289280 19536 heartbeater.cc:511] Master 127.10.224.190:44809 requested a full tablet report, sending...
I20260926 12:39:06.291241 19342 ts_manager.cc:194] Registered new tserver with Master: 482cf573ff544526b11458828f56efe7 (127.10.224.129:45647)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:39:06.293044 19342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:33299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:39:06.311951 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44809
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:06.431888 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:06.432175 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:06.432220 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:06.436470 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:06.436544 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:06.436642 19541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:39:06.442262 19541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:06.443456 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:06.444583 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:06.455329 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:06.455333 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:06.455425 19541 server_base.cc:1034] running on GCE node
W20260926 12:39:06.455363 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:06.456096 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:06.456794 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:06.458034 19541 hybrid_clock.cc:648] HybridClock initialized: now 1790426346457996 us; error 50 us; skew 500 ppm
Sep 26 12:39:06 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:06.461047 19541 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:39:06.462030 19541 webserver.cc:460] Webserver started at http://127.10.224.130:43237/ using document root <none> and password file <none>
I20260926 12:39:06.462623 19541 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:06.462687 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:06.462924 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:06.465152 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "f33226d1c0244831908716a445713c95"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "221775de43e3623faaa7afa952295a74"
server_key_iv: "c56f0ca0cd86b6115e1f5cbacad7ae14"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.465771 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "f33226d1c0244831908716a445713c95"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "221775de43e3623faaa7afa952295a74"
server_key_iv: "c56f0ca0cd86b6115e1f5cbacad7ae14"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.470086 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:39:06.472898 19556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.474234 19541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:39:06.474391 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "f33226d1c0244831908716a445713c95"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "221775de43e3623faaa7afa952295a74"
server_key_iv: "c56f0ca0cd86b6115e1f5cbacad7ae14"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.474519 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:06.494098 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:06.498055 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:06.498293 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:06.498941 19541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:06.499918 19541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:06.499996 19541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.500087 19541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:06.500138 19541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.509583 19541 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:46375
I20260926 12:39:06.509613 19669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:46375 every 8 connection(s)
I20260926 12:39:06.510560 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:39:06.519420 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 19541
I20260926 12:39:06.519642 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:39:06.520021 11138 external_mini_cluster.cc:1428] Setting key 083d5ff469c94815808d85837803705e
Sep 26 12:39:06 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:39:06.526816 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:06.512642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:49345 (local address 127.10.224.190:44809)
0926 12:39:06.512784 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:06.512788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:06.513813 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 12:39:06.515153 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:06.515157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:06.515221 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:06.515306 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:06.516718 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.517260 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.518618 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.518863 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.518985 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:06.522287 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:06.522295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:06.522300 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:06.522335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:06.524215 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:06.524998 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:06.525004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:06.525009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:06.525109 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:06.525483 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:06.525489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:06.525494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:06.525769 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:06.525887 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:06.526418 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 12:39:06.526582 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:39:06.527717 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:06.512986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:49345)
0926 12:39:06.513603 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:06.513651 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:06.514905 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:06.515504 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:06.515512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:06.515978 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:06.516528 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:06.516542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.517422 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.517425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:06.518479 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:06.518491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.519036 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.519040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:06.519228 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:06.519985 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:06.520004 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:06.522080 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:06.524362 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:06.524373 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:06.524398 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:06.524858 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:06.525217 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:06.525220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:06.525222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:06.525378 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:06.525872 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:06.525877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:06.526160 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 12:39:06.526326 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 12:39:06.526695 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":122,"threads_started":1}
I20260926 12:39:06.529093 19670 heartbeater.cc:347] Connected to a master server at 127.10.224.190:44809
I20260926 12:39:06.529404 19670 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:06.529953 19670 heartbeater.cc:511] Master 127.10.224.190:44809 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:39:06.531286 19342 ts_manager.cc:194] Registered new tserver with Master: f33226d1c0244831908716a445713c95 (127.10.224.130:46375)
I20260926 12:39:06.532087 19342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:49345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:39:06.552803 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44809
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:06.673692 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:06.673990 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:06.674036 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:06.678211 19675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:06.678280 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:06.678342 19675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:39:06.683933 19675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:06.685048 19675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:06.686179 19675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:06.696297 19681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:06.696444 19675 server_base.cc:1034] running on GCE node
W20260926 12:39:06.696323 19683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:06.696318 19680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:06.696898 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:06.697624 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:06.698828 19675 hybrid_clock.cc:648] HybridClock initialized: now 1790426346698807 us; error 34 us; skew 500 ppm
Sep 26 12:39:06 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:06.701685 19675 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:39:06.702543 19675 webserver.cc:460] Webserver started at http://127.10.224.131:44323/ using document root <none> and password file <none>
I20260926 12:39:06.703153 19675 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:06.703245 19675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:06.703501 19675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:06.705642 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "545150213dbf44a184944c7101d15b22"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "147ec6e7d5dadea0b5b8f90039995ae1"
server_key_iv: "447b5c83094f27143342339ddfbe3b3d"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.706184 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "545150213dbf44a184944c7101d15b22"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "147ec6e7d5dadea0b5b8f90039995ae1"
server_key_iv: "447b5c83094f27143342339ddfbe3b3d"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.710325 19675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 12:39:06.713135 19690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.714521 19675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:39:06.714674 19675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "545150213dbf44a184944c7101d15b22"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "147ec6e7d5dadea0b5b8f90039995ae1"
server_key_iv: "447b5c83094f27143342339ddfbe3b3d"
server_key_version: "encryptionkey@0"
I20260926 12:39:06.714802 19675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:06.726872 19675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:06.730413 19675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:06.730633 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:06.731261 19675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:06.732149 19675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:06.732198 19675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.732239 19675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:06.732295 19675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:06.741448 19675 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:42881
I20260926 12:39:06.741489 19803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:42881 every 8 connection(s)
I20260926 12:39:06.742446 19675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:39:06.750484 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 19675
I20260926 12:39:06.750648 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:39:06.750980 11138 external_mini_cluster.cc:1428] Setting key 3e54eccdfff0f48a9f92d32a13b370cb
Sep 26 12:39:06 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:39:06.757164 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:06.744639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:39049 (local address 127.10.224.190:44809)
0926 12:39:06.744814 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:06.744819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:06.745955 (+ 1136us) server_negotiation.cc:407] Connection header received
0926 12:39:06.746779 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:06.746782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:06.746831 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:06.746908 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:06.748189 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.748727 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.749834 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.750089 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.750175 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:06.753341 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:06.753345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:06.753348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:06.753378 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:06.754905 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:06.755508 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:06.755512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:06.755514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:06.755629 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:06.756000 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:06.756003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:06.756005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:06.756203 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:06.756317 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:06.756866 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 12:39:06.756995 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 12:39:06.758008 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:06.745097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:39049)
0926 12:39:06.745777 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:06.745820 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:06.746592 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:06.747095 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:06.747101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:06.747488 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:06.748053 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:06.748067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.748835 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.748837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:06.749718 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:06.749726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:06.750283 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:06.750291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:06.750514 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:06.751222 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:06.751244 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:06.753170 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:06.755047 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:06.755054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:06.755072 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:06.755393 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:06.755726 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:06.755729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:06.755731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:06.755890 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:06.756314 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:06.756321 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:06.756608 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 12:39:06.756806 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 12:39:06.757148 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":112,"threads_started":1}
I20260926 12:39:06.759210 19804 heartbeater.cc:347] Connected to a master server at 127.10.224.190:44809
I20260926 12:39:06.759599 19804 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:06.760146 19804 heartbeater.cc:511] Master 127.10.224.190:44809 requested a full tablet report, sending...
I20260926 12:39:06.761229 19342 ts_manager.cc:194] Registered new tserver with Master: 545150213dbf44a184944c7101d15b22 (127.10.224.131:42881)
I20260926 12:39:06.761859 19342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:39049
I20260926 12:39:06.765360 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab.
Entry for principal oryx/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab.
WARNING: no policy specified for oryx/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab.
Entry for principal oryx/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab.
WARNING: no policy specified for oryx/127.10.224.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab.
Entry for principal oryx/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab.
I20260926 12:39:06.851042 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 19407
I20260926 12:39:06.856835 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 19541
I20260926 12:39:06.862656 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 19675
I20260926 12:39:06.868357 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 19313
I20260926 12:39:06.875000 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:44809
--webserver_interface=127.10.224.190
--webserver_port=43911
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:06.995223 19813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:06.995630 19813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:06.995745 19813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:06.999995 19813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:39:07.000109 19813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:07.000146 19813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:07.000186 19813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:39:07.000236 19813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:39:07.006279 19813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=43911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:07.007630 19813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:07.008805 19813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:07.017977 19818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:07.018002 19819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:07.018055 19813 server_base.cc:1034] running on GCE node
W20260926 12:39:07.017987 19821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:07.019078 19813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:07.020579 19813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:07.021823 19813 hybrid_clock.cc:648] HybridClock initialized: now 1790426347021799 us; error 42 us; skew 500 ppm
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:07.025285 19813 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:39:07.026399 19813 webserver.cc:460] Webserver started at http://127.10.224.190:43911/ using document root <none> and password file <none>
I20260926 12:39:07.027031 19813 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:07.027148 19813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:07.031411 19813 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:07.033924 19828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.035404 19813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 12:39:07.035615 19813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "581c6c9b2b1f4b1f9726d929da1557ef"
format_stamp: "Formatted at 2026-09-26 12:39:05 on dist-test-slave-m3pj"
server_key: "f86752f11228ee8f4bc2d32769cf7c39"
server_key_iv: "7a9f694936969c5bc6f6785a9f25045f"
server_key_version: "encryptionkey@0"
I20260926 12:39:07.036266 19813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:07.059935 19813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:07.063668 19813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:07.063901 19813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:07.071856 19813 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:44809
I20260926 12:39:07.071887 19880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:44809 every 8 connection(s)
I20260926 12:39:07.073037 19813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:39:07.077680 19881 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:39:07.079706 19881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Bootstrap starting.
I20260926 12:39:07.082772 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 19813
I20260926 12:39:07.083274 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:45647
--local_ip_for_outbound_sockets=127.10.224.129
--tserver_master_addrs=127.10.224.190:44809
--webserver_port=42913
--webserver_interface=127.10.224.129
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.224.129
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:39:07.098857 19881 log.cc:824] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:07.106293 19881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:39:07.106730 19881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Bootstrap complete.
I20260926 12:39:07.111667 19881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } }
I20260926 12:39:07.112150 19881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581c6c9b2b1f4b1f9726d929da1557ef, State: Initialized, Role: FOLLOWER
I20260926 12:39:07.112725 19881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } }
I20260926 12:39:07.112879 19881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:39:07.112974 19881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:39:07.113081 19881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:39:07.115285 19881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } }
I20260926 12:39:07.115724 19881 leader_election.cc:302] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 581c6c9b2b1f4b1f9726d929da1557ef; no voters:
I20260926 12:39:07.116093 19881 leader_election.cc:288] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:39:07.116189 19885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:39:07.116389 19885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [term 2 LEADER]: Becoming Leader. State: Replica: 581c6c9b2b1f4b1f9726d929da1557ef, State: Running, Role: LEADER
I20260926 12:39:07.116773 19885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } }
I20260926 12:39:07.117446 19881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:39:07.118239 19887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 581c6c9b2b1f4b1f9726d929da1557ef. Latest consensus state: current_term: 2 leader_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } } }
I20260926 12:39:07.118386 19887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:07.118783 19886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581c6c9b2b1f4b1f9726d929da1557ef" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44809 } } }
I20260926 12:39:07.118875 19886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:07.119272 19892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:39:07.122850 19892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:39:07.123678 19892 catalog_manager.cc:1259] Loaded cluster ID: a9e8359cd7ba4f328af147c9a66099db
I20260926 12:39:07.123747 19892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:39:07.126217 19892 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:39:07.127069 19892 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 581c6c9b2b1f4b1f9726d929da1557ef: Loaded TSK: 0
I20260926 12:39:07.128475 19892 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:39:07.208458 19883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:07.209002 19883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:07.209132 19883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:07.213930 19883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:07.214046 19883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:07.214170 19883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:39:07.220717 19883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.129.keytab
--principal=oryx/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:45647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=42913
--tserver_master_addrs=127.10.224.190:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:07.221971 19883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:07.223093 19883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:07.233951 19908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:07.233954 19907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:07.234069 19910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:07.234436 19883 server_base.cc:1034] running on GCE node
I20260926 12:39:07.235005 19883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:07.236100 19883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:07.237361 19883 hybrid_clock.cc:648] HybridClock initialized: now 1790426347237307 us; error 65 us; skew 500 ppm
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426347, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:07.241860 19883 init.cc:377] Logged in from keytab as oryx/127.10.224.129@KRBTEST.COM (short username oryx)
I20260926 12:39:07.243183 19883 webserver.cc:460] Webserver started at http://127.10.224.129:42913/ using document root <none> and password file <none>
I20260926 12:39:07.243993 19883 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:07.244125 19883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:07.250663 19883 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 12:39:07.263185 19917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.265748 19883 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260926 12:39:07.265942 19883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "482cf573ff544526b11458828f56efe7"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "9304087e2d03aac586a7a4587c280a56"
server_key_iv: "47d47649716dd2b9262f5d369fd8bfc1"
server_key_version: "encryptionkey@0"
I20260926 12:39:07.266705 19883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:07.288149 19883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:07.291919 19883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:07.292146 19883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:07.292822 19883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:07.293749 19883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:07.293799 19883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.293867 19883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:07.293900 19883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.303566 19883 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:45647
I20260926 12:39:07.303659 20030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:45647 every 8 connection(s)
I20260926 12:39:07.304723 19883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:39:07.313660 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 19883
I20260926 12:39:07.314203 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:46375
--local_ip_for_outbound_sockets=127.10.224.130
--tserver_master_addrs=127.10.224.190:44809
--webserver_port=43237
--webserver_interface=127.10.224.130
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.224.130
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:07.320564 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.307071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:59605)
0926 12:39:07.307816 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.307872 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.308685 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.309846 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.309854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.310372 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.310883 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.310899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.312836 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.312840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.314189 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.314205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.315223 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.315230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.315492 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.316223 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.316241 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.319552 (+ 3311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":128,"threads_started":1}
W20260926 12:39:07.321027 20033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.306901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:59605 (local address 127.10.224.190:44809)
0926 12:39:07.307374 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.307389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.307989 (+ 600us) server_negotiation.cc:407] Connection header received
0926 12:39:07.309062 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.309075 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.309383 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.309709 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.311111 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.312654 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.314373 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.314998 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.315186 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.319936 (+ 4750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:59605: BlockingRecv error: recv got EOF from 127.10.224.129:59605 (error 108)
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":130,"threads_started":1}
W20260926 12:39:07.321106 20031 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:44809 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.224.190:44809: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:07.328907 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.321936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:43905)
0926 12:39:07.322156 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.322175 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.322291 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.322595 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.322600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.322821 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.322993 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.323004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.324049 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.324057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.325243 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.325260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.325784 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.325789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.325932 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.326504 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.326524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.328719 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260926 12:39:07.329015 20033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.322051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:43905 (local address 127.10.224.190:44809)
0926 12:39:07.322175 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.322179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.322227 (+ 48us) server_negotiation.cc:407] Connection header received
0926 12:39:07.322414 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.322416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.322463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.322547 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.323161 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.323887 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.325415 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.325643 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.325746 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.328893 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:43905: BlockingRecv error: recv got EOF from 127.10.224.129:43905 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:07.335484 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.329609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:34799)
0926 12:39:07.329732 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.329748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.329849 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.330179 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.330181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.330335 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.330426 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.330432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.331611 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.331615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.332167 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.332176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.332757 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.332759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.332827 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.333495 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.333508 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.335376 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 12:39:07.335719 20033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.329691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:34799 (local address 127.10.224.190:44809)
0926 12:39:07.329819 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.329824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.329848 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:39:07.329949 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.329952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.330003 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.330096 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.330547 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.331390 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.332287 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.332638 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.332744 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.335596 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:34799: BlockingRecv error: recv got EOF from 127.10.224.129:34799 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 12:39:07.335940 20031 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:39:07.440071 20035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:07.440439 20035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:07.440521 20035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:07.444962 20035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:07.445043 20035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:07.445142 20035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:39:07.450910 20035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.130.keytab
--principal=oryx/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:46375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=43237
--tserver_master_addrs=127.10.224.190:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:07.452576 20035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:07.453748 20035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:07.464605 20040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:07.464768 20035 server_base.cc:1034] running on GCE node
W20260926 12:39:07.464618 20041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:07.464638 20043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:07.465358 20035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:07.466082 20035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:07.467260 20035 hybrid_clock.cc:648] HybridClock initialized: now 1790426347467222 us; error 49 us; skew 500 ppm
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426347, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:07.470779 20035 init.cc:377] Logged in from keytab as oryx/127.10.224.130@KRBTEST.COM (short username oryx)
I20260926 12:39:07.471962 20035 webserver.cc:460] Webserver started at http://127.10.224.130:43237/ using document root <none> and password file <none>
I20260926 12:39:07.472637 20035 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:07.472751 20035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:07.477272 20035 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:39:07.479995 20050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.481581 20035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:39:07.481776 20035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "f33226d1c0244831908716a445713c95"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "221775de43e3623faaa7afa952295a74"
server_key_iv: "c56f0ca0cd86b6115e1f5cbacad7ae14"
server_key_version: "encryptionkey@0"
I20260926 12:39:07.482429 20035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:07.499924 20035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:07.503862 20035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:07.504122 20035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:07.504849 20035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:07.505955 20035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:07.506044 20035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.506119 20035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:07.506168 20035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.516178 20035 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:46375
I20260926 12:39:07.516201 20163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:46375 every 8 connection(s)
I20260926 12:39:07.517251 20035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:39:07.522836 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 20035
I20260926 12:39:07.523340 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:42881
--local_ip_for_outbound_sockets=127.10.224.131
--tserver_master_addrs=127.10.224.190:44809
--webserver_port=44323
--webserver_interface=127.10.224.131
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.224.131
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:39:07.529851 20033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.519276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:49277 (local address 127.10.224.190:44809)
0926 12:39:07.519636 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.519641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.521010 (+ 1369us) server_negotiation.cc:407] Connection header received
0926 12:39:07.521891 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.521894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.521947 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.522027 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.523680 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.524208 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.525531 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.525757 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.525875 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.529725 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:49277: BlockingRecv error: recv got EOF from 127.10.224.130:49277 (error 108)
Metrics: {"server-negotiator.queue_time_us":188}
I20260926 12:39:07.530535 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.519835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:49277)
0926 12:39:07.520802 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.520885 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.521684 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.522202 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.522209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.522693 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.523433 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.523458 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.524351 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.524355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.525395 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.525405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.525925 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.525929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.526103 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.526793 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.526819 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.529589 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":548,"threads_started":1}
W20260926 12:39:07.531061 20164 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:44809 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.224.190:44809: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:07.538163 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.531592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:49983)
0926 12:39:07.531813 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.531829 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.531951 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.532216 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.532221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.532424 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.532577 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.532588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.533543 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.533550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.534592 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.534604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.535075 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.535079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.535175 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.535814 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.535828 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.538018 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120}
W20260926 12:39:07.538357 20033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.531682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:49983 (local address 127.10.224.190:44809)
0926 12:39:07.531800 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.531804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.531892 (+ 88us) server_negotiation.cc:407] Connection header received
0926 12:39:07.532051 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.532053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.532094 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.532166 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.532711 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.533224 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.534734 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.534959 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.535053 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.538180 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:49983: BlockingRecv error: recv got EOF from 127.10.224.130:49983 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:07.544441 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.538848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:37551)
0926 12:39:07.539008 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.539021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.539090 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.539318 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.539321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.539481 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.539592 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.539598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.540470 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.540473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.541398 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.541409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.541843 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.541847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.541937 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.542417 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.542431 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.544330 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260926 12:39:07.544765 20033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.538950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:37551 (local address 127.10.224.190:44809)
0926 12:39:07.539072 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.539076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.539090 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:39:07.539172 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.539174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.539215 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.539289 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.539702 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.540370 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.541514 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.541736 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.541825 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.544500 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:37551: BlockingRecv error: recv got EOF from 127.10.224.130:37551 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 12:39:07.544780 20164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:39:07.649879 20167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:07.650172 20167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:07.650216 20167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:07.654372 20167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:07.654464 20167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:07.654529 20167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:39:07.660288 20167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/krb5kdc/oryx_127.10.224.131.keytab
--principal=oryx/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:42881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=44323
--tserver_master_addrs=127.10.224.190:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:07.661471 20167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:07.662585 20167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:07.672992 20173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:07.672992 20172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:07.672992 20175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:07.673254 20167 server_base.cc:1034] running on GCE node
I20260926 12:39:07.673870 20167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:07.674602 20167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:07.675793 20167 hybrid_clock.cc:648] HybridClock initialized: now 1790426347675753 us; error 55 us; skew 500 ppm
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426347, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:07.678897 20167 init.cc:377] Logged in from keytab as oryx/127.10.224.131@KRBTEST.COM (short username oryx)
I20260926 12:39:07.680002 20167 webserver.cc:460] Webserver started at http://127.10.224.131:44323/ using document root <none> and password file <none>
I20260926 12:39:07.680639 20167 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:07.680718 20167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:07.684948 20167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:39:07.687563 20182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.689108 20167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:07.689275 20167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "545150213dbf44a184944c7101d15b22"
format_stamp: "Formatted at 2026-09-26 12:39:06 on dist-test-slave-m3pj"
server_key: "147ec6e7d5dadea0b5b8f90039995ae1"
server_key_iv: "447b5c83094f27143342339ddfbe3b3d"
server_key_version: "encryptionkey@0"
I20260926 12:39:07.690001 20167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:07.699734 20167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:07.703033 20167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:07.703260 20167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:07.704042 20167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:07.704965 20167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:07.705039 20167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.705111 20167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:07.705164 20167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:07.715862 20167 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:42881
I20260926 12:39:07.715956 20295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:42881 every 8 connection(s)
I20260926 12:39:07.716998 20167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:39:07.720281 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 20167
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:39:07.730672 20033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.719162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:46817 (local address 127.10.224.190:44809)
0926 12:39:07.719327 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.719332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.720986 (+ 1654us) server_negotiation.cc:407] Connection header received
0926 12:39:07.722246 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.722249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.722312 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.722410 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.723816 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.724387 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.725941 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.726199 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.726347 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.730544 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:46817: BlockingRecv error: recv got EOF from 127.10.224.131:46817 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:39:07.731511 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.719774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:46817)
0926 12:39:07.720767 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.720863 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.721995 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.722608 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.722616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.723025 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.723645 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.723662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.724544 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.724548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.725798 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.725811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.726407 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.726412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.726618 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.727296 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.727322 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.730375 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":199,"threads_started":1}
I20260926 12:39:07.731818 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.721635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.10.224.190:44809)
0926 12:39:07.721923 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.721927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.721939 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:39:07.721982 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.721984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.722025 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.722141 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.722960 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.723655 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.724629 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.724899 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.725030 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.726826 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:07.726834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:07.726852 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:07.726887 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:07.729405 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:07.729986 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:07.729991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:07.729997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:07.730127 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:07.730426 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:07.730430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:07.730432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:07.730881 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:07.731058 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:07.731351 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 12:39:07.731689 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
W20260926 12:39:07.732491 20296 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:44809 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.224.190:44809: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:07.738570 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.732968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:43043)
0926 12:39:07.733085 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.733100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.733218 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.733469 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.733472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.733662 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.734066 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.734076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.734815 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.734818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.735431 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.735440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.735970 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.735973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.736046 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.736582 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.736594 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.738457 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 12:39:07.738874 20300 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.733010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:43043 (local address 127.10.224.190:44809)
0926 12:39:07.733109 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.733115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.733168 (+ 53us) server_negotiation.cc:407] Connection header received
0926 12:39:07.733319 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.733322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.733363 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.733453 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.734166 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.734711 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.735619 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.735844 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.735952 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.738642 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:43043: BlockingRecv error: recv got EOF from 127.10.224.131:43043 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:07 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:07.744385 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:07.739161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:43705)
0926 12:39:07.739303 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:07.739320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:07.739412 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:07.739749 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:07.739755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:07.739928 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:07.740051 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.740058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.740820 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.740823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.741364 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:07.741373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.741794 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.741797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:07.741873 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.742355 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:07.742371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:07.744254 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 12:39:07.744627 20296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:39:07.744694 20300 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:07.739208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:43705 (local address 127.10.224.190:44809)
0926 12:39:07.739382 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:07.739387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:07.739399 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:39:07.739496 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:07.739499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:07.739580 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:07.739665 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:07.740159 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.740704 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.741476 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:07.741697 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:07.741780 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:07.744578 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:43705: BlockingRecv error: recv got EOF from 127.10.224.131:43705 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 12:39:08 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:08 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:08.342422 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:08.336653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:32883)
0926 12:39:08.336933 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:08.336946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:08.337051 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:08.337302 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:08.337303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:08.337484 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:08.337602 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:08.337607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.338436 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.338438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:08.339029 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:08.339039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.339606 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.339609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:08.339682 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:08.340286 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:08.340299 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:08.342294 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
W20260926 12:39:08.342592 20302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:08.336729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:32883 (local address 127.10.224.190:44809)
0926 12:39:08.337029 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:08.337033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:08.337047 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:39:08.337137 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:08.337140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:08.337186 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:08.337273 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:08.337720 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.338315 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.339173 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.339416 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.339511 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:08.342447 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:32883: BlockingRecv error: recv got EOF from 127.10.224.129:32883 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
Sep 26 12:39:08 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:08 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:08.551075 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:08.545514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:51471)
0926 12:39:08.545834 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:08.545847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:08.545959 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:08.546201 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:08.546204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:08.546382 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:08.546484 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:08.546491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.547277 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.547279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:08.547957 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:08.547967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.548391 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.548394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:08.548455 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:08.549055 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:08.549068 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:08.550941 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
W20260926 12:39:08.551288 20302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:08.545582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:51471 (local address 127.10.224.190:44809)
0926 12:39:08.545740 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:08.545746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:08.545921 (+ 175us) server_negotiation.cc:407] Connection header received
0926 12:39:08.546045 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:08.546047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:08.546093 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:08.546165 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:08.546610 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.547169 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.548073 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.548286 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.548382 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:08.551143 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:51471: BlockingRecv error: recv got EOF from 127.10.224.130:51471 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 12:39:08 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:08 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:08.751420 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:08.745459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:32847)
0926 12:39:08.745784 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:08.745798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:08.745904 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:08.746141 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:08.746143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:08.746327 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:08.746498 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:08.746505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.747279 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.747282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:08.747977 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:08.747987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.748589 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.748592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:08.748697 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:08.749319 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:08.749335 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:08.751245 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
W20260926 12:39:08.751509 20302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:08.745590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:32847 (local address 127.10.224.190:44809)
0926 12:39:08.745783 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:08.745787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:08.745855 (+ 68us) server_negotiation.cc:407] Connection header received
0926 12:39:08.745984 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:08.745986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:08.746030 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:08.746103 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:08.746626 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.747162 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.748106 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:08.748455 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:08.748581 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:08.751390 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:32847: BlockingRecv error: recv got EOF from 127.10.224.131:32847 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 12:39:09 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:09 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:09.351435 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:09.343691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:36205)
0926 12:39:09.344073 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:09.344095 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:09.344273 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:09.344718 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:09.344721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:09.344955 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:09.345132 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:09.345139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.346027 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.346031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:09.346686 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:09.346698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.347437 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.347441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:09.347606 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:09.348447 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:09.348471 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:09.351203 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
W20260926 12:39:09.351742 20306 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:09.343761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:36205 (local address 127.10.224.190:44809)
0926 12:39:09.344089 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:09.344093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:09.344248 (+ 155us) server_negotiation.cc:407] Connection header received
0926 12:39:09.344397 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:09.344400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:09.344464 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:09.344611 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:09.345283 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.345858 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.346874 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.347174 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.347356 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:09.351457 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:36205: BlockingRecv error: recv got EOF from 127.10.224.129:36205 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":64,"threads_started":1}
Sep 26 12:39:09 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:09 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:09.567709 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:09.552421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:42085)
0926 12:39:09.552830 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:09.552848 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:09.553007 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:09.553342 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:09.553345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:09.553583 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:09.553722 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:09.553727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.554731 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.554735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:09.555413 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:09.555422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.556101 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.556107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:09.556230 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:09.564959 (+ 8729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:09.564983 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:09.567424 (+ 2441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
W20260926 12:39:09.568022 20306 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:09.552700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:42085 (local address 127.10.224.190:44809)
0926 12:39:09.552940 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:09.552945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:09.552966 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:39:09.553136 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:09.553139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:09.553192 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:09.553280 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:09.553914 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.554574 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.555591 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.555930 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.556052 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:09.567810 (+ 11758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:42085: BlockingRecv error: recv got EOF from 127.10.224.130:42085 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 26 12:39:09 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:09 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:09.759948 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:09.752498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:55417)
0926 12:39:09.752913 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:09.752929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:09.753088 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:09.753402 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:09.753404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:09.753641 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:09.753755 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:09.753761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.754697 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.754700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:09.755351 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:09.755363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.755977 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.755980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:09.756073 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:09.757080 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:09.757101 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:09.759769 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
W20260926 12:39:09.760165 20306 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:09.752773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:55417 (local address 127.10.224.190:44809)
0926 12:39:09.752942 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:09.752951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:09.753012 (+ 61us) server_negotiation.cc:407] Connection header received
0926 12:39:09.753202 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:09.753204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:09.753264 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:09.753344 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:09.753904 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.754540 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.755476 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:09.755816 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:09.755946 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:09.759956 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:55417: BlockingRecv error: recv got EOF from 127.10.224.131:55417 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:39:10 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:10 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:10.358915 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:10.352469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:50819)
0926 12:39:10.352766 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:10.352780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:10.352877 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:10.353202 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:10.353204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:10.353392 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:10.353508 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:10.353514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.354353 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.354356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:10.355012 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:10.355023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.355691 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.355694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:10.355763 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:10.356365 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:10.356377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:10.358768 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
W20260926 12:39:10.359140 20310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:10.352560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:50819 (local address 127.10.224.190:44809)
0926 12:39:10.352839 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:10.352845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:10.352860 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:39:10.352953 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:10.352956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:10.353005 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:10.353108 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:10.353627 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.354239 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.355184 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.355464 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.355676 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:10.358966 (+ 3290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:50819: BlockingRecv error: recv got EOF from 127.10.224.129:50819 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
Sep 26 12:39:10 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:10 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:10.575353 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:10.568928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:54995)
0926 12:39:10.569447 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:10.569466 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:10.569597 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:10.569900 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:10.569903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:10.570135 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:10.570245 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:10.570252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.571084 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.571089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:10.571766 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:10.571776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.572238 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.572242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:10.572322 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:10.572890 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:10.572904 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:10.575171 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":106,"threads_started":1}
W20260926 12:39:10.575443 20310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:10.569057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:54995 (local address 127.10.224.190:44809)
0926 12:39:10.569206 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:10.569212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:10.569609 (+ 397us) server_negotiation.cc:407] Connection header received
0926 12:39:10.569691 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:10.569694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:10.569760 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:10.569882 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:10.570373 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.570966 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.571895 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.572119 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.572235 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:10.575334 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:54995: BlockingRecv error: recv got EOF from 127.10.224.130:54995 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 12:39:10 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:10 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:10.767768 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:10.761143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:50703)
0926 12:39:10.761456 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:10.761473 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:10.761601 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:10.761888 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:10.761891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:10.762133 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:10.762283 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:10.762293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.763313 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.763317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:10.764160 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:10.764172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.764729 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.764732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:10.764804 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:10.765487 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:10.765508 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:10.767620 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":124,"threads_started":1}
W20260926 12:39:10.767966 20310 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:10.761259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:50703 (local address 127.10.224.190:44809)
0926 12:39:10.761586 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:10.761590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:10.761610 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:39:10.761681 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:10.761684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:10.761733 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:10.761822 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:10.762432 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.763138 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.764302 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:10.764589 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:10.764689 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:10.767782 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:50703: BlockingRecv error: recv got EOF from 127.10.224.131:50703 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Sep 26 12:39:11 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:11 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:11.366945 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:11.360235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:60445)
0926 12:39:11.360524 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:11.360537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:11.360648 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:11.360979 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:11.360982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:11.361176 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:11.361286 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:11.361291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.362488 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.362493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:11.363283 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:11.363298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.363871 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.363874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:11.363944 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:11.364597 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:11.364611 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:11.366806 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
W20260926 12:39:11.367179 20314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:11.360387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:60445 (local address 127.10.224.190:44809)
0926 12:39:11.360699 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:11.360704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:11.360719 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:39:11.360765 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:11.360768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:11.360813 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:11.360927 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:11.361471 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.362289 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.363443 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.363751 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.363847 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:11.367016 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:60445: BlockingRecv error: recv got EOF from 127.10.224.129:60445 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":138,"threads_started":1}
Sep 26 12:39:11 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:11 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:11.582777 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:11.576477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:49585)
0926 12:39:11.576765 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:11.576791 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:11.576909 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:11.577236 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:11.577238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:11.577504 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:11.577607 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:11.577613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.578458 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.578460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:11.579035 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:11.579045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.579679 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.579683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:11.579771 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:11.580415 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:11.580428 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:11.582643 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
W20260926 12:39:11.583012 20314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:11.576656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:49585 (local address 127.10.224.190:44809)
0926 12:39:11.576802 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:11.576806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:11.576872 (+ 66us) server_negotiation.cc:407] Connection header received
0926 12:39:11.577007 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:11.577009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:11.577057 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:11.577147 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:11.577744 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.578274 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.579198 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.579423 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.579563 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:11.582835 (+ 3272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:49585: BlockingRecv error: recv got EOF from 127.10.224.130:49585 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 12:39:11 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:11 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:11.775266 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:11.768729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:34959)
0926 12:39:11.769131 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:11.769153 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:11.769301 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:11.769659 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:11.769662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:11.769874 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:11.769972 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:11.769979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.770786 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.770801 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:11.771368 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:11.771378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.772031 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.772035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:11.772129 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:11.772775 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:11.772788 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:11.775077 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":151,"threads_started":1}
W20260926 12:39:11.775365 20314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:11.768910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:34959 (local address 127.10.224.190:44809)
0926 12:39:11.769092 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:11.769097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:11.769235 (+ 138us) server_negotiation.cc:407] Connection header received
0926 12:39:11.769413 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:11.769416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:11.769532 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:11.769636 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:11.770101 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.770656 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.771550 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:11.771877 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:11.772000 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:11.775238 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:34959: BlockingRecv error: recv got EOF from 127.10.224.131:34959 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 12:39:12 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:12 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:12.374138 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:12.367806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:45227)
0926 12:39:12.368139 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:12.368154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:12.368289 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:12.368657 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:12.368659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:12.368883 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:12.368979 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:12.368985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.369795 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.369798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:12.370389 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:12.370398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.370916 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.370919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:12.370990 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:12.371749 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:12.371771 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:12.373985 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
W20260926 12:39:12.374367 20318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:12.367961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:45227 (local address 127.10.224.190:44809)
0926 12:39:12.368215 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:12.368222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:12.368238 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:39:12.368422 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:12.368460 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:12.368508 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:12.368589 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:12.369093 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.369651 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.370515 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.370760 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.370871 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:12.374171 (+ 3300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:45227: BlockingRecv error: recv got EOF from 127.10.224.129:45227 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
Sep 26 12:39:12 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:12 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:12.590091 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:12.583786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:59207)
0926 12:39:12.584148 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:12.584171 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:12.584264 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:12.584596 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:12.584599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:12.584795 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:12.584914 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:12.584919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.585750 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.585753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:12.586618 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:12.586632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.587208 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.587212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:12.587286 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:12.587881 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:12.587898 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:12.589971 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":123,"threads_started":1}
W20260926 12:39:12.590286 20318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:12.584030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:59207 (local address 127.10.224.190:44809)
0926 12:39:12.584223 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:12.584227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:12.584243 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:39:12.584385 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:12.584387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:12.584431 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:12.584504 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:12.585045 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.585636 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.586754 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.587075 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.587177 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:12.590147 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:59207: BlockingRecv error: recv got EOF from 127.10.224.130:59207 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 12:39:12 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:12 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:12.783162 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:12.776321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:33965)
0926 12:39:12.776710 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:12.776724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:12.776832 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:12.777108 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:12.777110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:12.777302 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:12.777399 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:12.777405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.778577 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.778581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:12.779171 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:12.779182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.779900 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.779904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:12.779992 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:12.780660 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:12.780680 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:12.782994 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":210,"threads_started":1}
W20260926 12:39:12.783326 20318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:12.776504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:33965 (local address 127.10.224.190:44809)
0926 12:39:12.776652 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:12.776656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:12.776867 (+ 211us) server_negotiation.cc:407] Connection header received
0926 12:39:12.776924 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:12.776926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:12.776978 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:12.777092 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:12.777587 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.778398 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.779403 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:12.779750 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:12.779886 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:12.783162 (+ 3276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:33965: BlockingRecv error: recv got EOF from 127.10.224.131:33965 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 12:39:13 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:13 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:13.381665 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:13.375215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:50809)
0926 12:39:13.375615 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:13.375630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:13.375725 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:13.376012 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:13.376014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:13.376200 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:13.376296 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:13.376302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.377060 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.377063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:13.377620 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:13.377628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.378157 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.378160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:13.378238 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:13.379058 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:13.379072 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:13.381491 (+ 2419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
W20260926 12:39:13.381846 20322 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:13.375371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:50809 (local address 127.10.224.190:44809)
0926 12:39:13.375654 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:13.375658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:13.375693 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:39:13.375828 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:13.375830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:13.375879 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:13.375959 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:13.376407 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.376951 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.377743 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.377986 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.378095 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:13.381665 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:50809: BlockingRecv error: recv got EOF from 127.10.224.129:50809 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
Sep 26 12:39:13 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:13 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:13.597582 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:13.590983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:43169)
0926 12:39:13.591323 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:13.591341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:13.591464 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:13.591811 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:13.591814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:13.592123 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:13.592223 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:13.592229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.593033 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.593038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:13.593787 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:13.593798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.594280 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.594283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:13.594360 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:13.594995 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:13.595008 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:13.597410 (+ 2402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":95,"threads_started":1}
W20260926 12:39:13.597718 20322 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:13.591105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:43169 (local address 127.10.224.190:44809)
0926 12:39:13.591299 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:13.591304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:13.591423 (+ 119us) server_negotiation.cc:407] Connection header received
0926 12:39:13.591633 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:13.591636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:13.591698 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:13.591790 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:13.592368 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.592914 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.593922 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.594158 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.594273 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:13.597604 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:43169: BlockingRecv error: recv got EOF from 127.10.224.130:43169 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 12:39:13 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:13 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:13.792436 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:13.784211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:42255)
0926 12:39:13.784507 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:13.784533 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:13.784634 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:13.784963 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:13.784966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:13.785162 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:13.785260 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:13.785266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.786600 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.786603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:13.787603 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:13.787616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.788510 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.788522 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:13.788752 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:13.789422 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:13.789440 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:13.792288 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":137,"threads_started":1}
W20260926 12:39:13.792732 20322 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:13.784453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:42255 (local address 127.10.224.190:44809)
0926 12:39:13.784609 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:13.784613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:13.784638 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:39:13.784713 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:13.784716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:13.784764 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:13.784884 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:13.785502 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.786410 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.787726 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:13.788149 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:13.788284 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:13.792546 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:42255: BlockingRecv error: recv got EOF from 127.10.224.131:42255 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 12:39:14 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:14 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:14.389272 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:14.382637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:47137)
0926 12:39:14.383047 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:14.383063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:14.383163 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:14.383542 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:14.383545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:14.383774 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:14.383894 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:14.383901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.384998 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.385003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:14.385718 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:14.385730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.386202 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.386204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:14.386265 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:14.386879 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:14.386903 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:14.389141 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":125,"threads_started":1}
W20260926 12:39:14.389663 20326 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:14.382820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:47137 (local address 127.10.224.190:44809)
0926 12:39:14.383101 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:14.383105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:14.383121 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:39:14.383300 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:14.383303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:14.383351 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:14.383449 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:14.384044 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.384862 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.385848 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.386094 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.386188 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:14.389506 (+ 3318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:47137: BlockingRecv error: recv got EOF from 127.10.224.129:47137 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
Sep 26 12:39:14 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:14 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:14.611387 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:14.598599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:59681)
0926 12:39:14.598963 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:14.598978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:14.599087 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:14.599389 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:14.599392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:14.599660 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:14.599760 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:14.599766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.600558 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.600561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:14.601140 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:14.601149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.601634 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.601637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:14.601706 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:14.608895 (+ 7189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:14.608914 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:14.611241 (+ 2327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
W20260926 12:39:14.611696 20326 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:14.598759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:59681 (local address 127.10.224.190:44809)
0926 12:39:14.598914 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:14.598918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:14.599094 (+ 176us) server_negotiation.cc:407] Connection header received
0926 12:39:14.599173 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:14.599175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:14.599224 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:14.599315 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:14.599895 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.600422 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.601253 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.601488 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.601600 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:14.611567 (+ 9967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:59681: BlockingRecv error: recv got EOF from 127.10.224.130:59681 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 12:39:14 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:14 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:14.799705 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:14.793457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:36711)
0926 12:39:14.793836 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:14.793850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:14.793968 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:14.794384 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:14.794387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:14.794593 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:14.794698 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:14.794705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.795561 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.795563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:14.796140 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:14.796148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.796693 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.796697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:14.796805 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:14.797412 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:14.797424 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:14.799508 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":74,"threads_started":1}
W20260926 12:39:14.799975 20326 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:14.793635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:36711 (local address 127.10.224.190:44809)
0926 12:39:14.793818 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:14.793823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:14.793960 (+ 137us) server_negotiation.cc:407] Connection header received
0926 12:39:14.794145 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:14.794148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:14.794202 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:14.794361 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:14.794840 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.795430 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.796252 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:14.796517 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:14.796606 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:14.799821 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:36711: BlockingRecv error: recv got EOF from 127.10.224.131:36711 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 12:39:15 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:15 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:15.398023 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:15.390478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:45739)
0926 12:39:15.391084 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:15.391101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:15.391238 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:15.391639 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:15.391643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:15.391899 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:15.392007 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:15.392013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.392923 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.392925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:15.394082 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:15.394097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.394700 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.394704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:15.394794 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:15.395497 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:15.395513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:15.397793 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":335,"threads_started":1}
W20260926 12:39:15.398145 20330 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:15.390925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:45739 (local address 127.10.224.190:44809)
0926 12:39:15.391197 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:15.391201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:15.391218 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:39:15.391349 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:15.391352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:15.391401 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:15.391506 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:15.392146 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.392727 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.394222 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.394543 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.394679 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:15.397987 (+ 3308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:45739: BlockingRecv error: recv got EOF from 127.10.224.129:45739 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
Sep 26 12:39:15 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:15 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:15.619223 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:15.612414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:54729)
0926 12:39:15.612766 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:15.612781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:15.612936 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:15.613309 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:15.613313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:15.613568 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:15.613680 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:15.613687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.614729 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.614731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:15.615355 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:15.615365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.615882 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.615885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:15.615954 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:15.616519 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:15.616533 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:15.619081 (+ 2548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
W20260926 12:39:15.619455 20330 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:15.612570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:54729 (local address 127.10.224.190:44809)
0926 12:39:15.612719 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:15.612732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:15.612880 (+ 148us) server_negotiation.cc:407] Connection header received
0926 12:39:15.613087 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:15.613090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:15.613148 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:15.613243 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:15.613820 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.614604 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.615493 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.615765 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.615868 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:15.619304 (+ 3436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:54729: BlockingRecv error: recv got EOF from 127.10.224.130:54729 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:39:15 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:15 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:15.806854 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:15.800827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:50157)
0926 12:39:15.801114 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:15.801127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:15.801227 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:15.801520 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:15.801522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:15.801703 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:15.801803 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:15.801808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.802613 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.802616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:15.803167 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:15.803175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.803741 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.803744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:15.803803 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:15.804434 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:15.804446 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:15.806717 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260926 12:39:15.807088 20330 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:15.800929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:50157 (local address 127.10.224.190:44809)
0926 12:39:15.801140 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:15.801144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:15.801187 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:39:15.801327 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:15.801329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:15.801423 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:15.801509 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:15.801918 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.802489 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.803280 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:15.803637 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:15.803722 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:15.806861 (+ 3139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:50157: BlockingRecv error: recv got EOF from 127.10.224.131:50157 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 12:39:16 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:16 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:16.406133 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:16.399184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:57183)
0926 12:39:16.399511 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:16.399555 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:16.399655 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:16.399927 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:16.399930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:16.400120 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:16.400212 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:16.400217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.401154 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.401157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:16.401789 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:16.401798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.402282 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.402284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:16.402349 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:16.403091 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:16.403114 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:16.405964 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":88,"threads_started":1}
W20260926 12:39:16.406613 20334 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:16.399280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:57183 (local address 127.10.224.190:44809)
0926 12:39:16.399655 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:16.399659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:16.399680 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:39:16.399726 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:16.399728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:16.399771 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:16.399862 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:16.400394 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.400993 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.401919 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.402165 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.402267 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:16.406471 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:57183: BlockingRecv error: recv got EOF from 127.10.224.129:57183 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":82,"threads_started":1}
Sep 26 12:39:16 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:16 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:16.626374 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:16.620377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:56035)
0926 12:39:16.620687 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:16.620701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:16.620811 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:16.621077 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:16.621079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:16.621261 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:16.621372 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:16.621377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.622163 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.622165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:16.622758 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:16.622767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.623233 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.623236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:16.623302 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:16.623856 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:16.623868 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:16.626235 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":124,"threads_started":1}
W20260926 12:39:16.626519 20334 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:16.620390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:56035 (local address 127.10.224.190:44809)
0926 12:39:16.620568 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:16.620573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:16.620845 (+ 272us) server_negotiation.cc:407] Connection header received
0926 12:39:16.620902 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:16.620904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:16.620950 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:16.621040 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:16.621514 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.622051 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.622895 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.623116 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.623201 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:16.626386 (+ 3185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:56035: BlockingRecv error: recv got EOF from 127.10.224.130:56035 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 12:39:16 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:16 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:16.813575 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:16.807783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:39545)
0926 12:39:16.808078 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:16.808095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:16.808208 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:16.808550 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:16.808556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:16.808790 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:16.808893 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:16.808899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.809683 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.809686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:16.810278 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:16.810286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.810760 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.810762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:16.810823 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:16.811341 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:16.811356 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:16.813450 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
W20260926 12:39:16.813766 20334 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:16.807838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:39545 (local address 127.10.224.190:44809)
0926 12:39:16.807968 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:16.807973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:16.808194 (+ 221us) server_negotiation.cc:407] Connection header received
0926 12:39:16.808335 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:16.808338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:16.808412 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:16.808497 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:16.809005 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.809545 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.810429 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:16.810650 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:16.810744 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:16.813616 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:39545: BlockingRecv error: recv got EOF from 127.10.224.131:39545 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 12:39:17 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:17 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:17.413641 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:17.407177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:45779)
0926 12:39:17.407547 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:17.407564 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:17.407677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:17.407993 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:17.407996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:17.408223 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:17.408364 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:17.408430 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.409490 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.409493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:17.410243 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:17.410253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.410892 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.410896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:17.410985 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:17.411542 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:17.411554 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:17.413473 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":80,"threads_started":1}
W20260926 12:39:17.413789 20338 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:17.407247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:45779 (local address 127.10.224.190:44809)
0926 12:39:17.407539 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:17.407547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:17.407647 (+ 100us) server_negotiation.cc:407] Connection header received
0926 12:39:17.407777 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:17.407780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:17.407843 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:17.407947 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:17.408564 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.409364 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.410371 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.410755 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.410875 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:17.413662 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:45779: BlockingRecv error: recv got EOF from 127.10.224.129:45779 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
Sep 26 12:39:17 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:17 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:17.632998 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:17.627354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:57141)
0926 12:39:17.627666 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:17.627686 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:17.627786 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:17.628073 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:17.628076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:17.628291 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:17.628381 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:17.628387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.629155 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.629160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:17.629731 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:17.629739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.630205 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.630208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:17.630293 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:17.630836 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:17.630848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:17.632827 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":82,"threads_started":1}
W20260926 12:39:17.633100 20338 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:17.627637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:57141 (local address 127.10.224.190:44809)
0926 12:39:17.627794 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:17.627799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:17.627818 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:39:17.627863 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:17.627868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:17.627919 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:17.628052 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:17.628504 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.629040 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.629855 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.630088 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.630201 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:17.632987 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:57141: BlockingRecv error: recv got EOF from 127.10.224.130:57141 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 12:39:17 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:17 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:17.821298 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:17.814727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:56213)
0926 12:39:17.815046 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:17.815060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:17.815143 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:17.815470 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:17.815473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:17.815704 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:17.815837 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:17.815843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.816621 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.816624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:17.817357 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:17.817367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.818244 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.818248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:17.818328 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:17.819033 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:17.819044 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:17.821109 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":127,"threads_started":1}
W20260926 12:39:17.821363 20338 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:17.815021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:56213 (local address 127.10.224.190:44809)
0926 12:39:17.815157 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:17.815161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:17.815175 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:39:17.815217 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:17.815220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:17.815261 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:17.815398 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:17.815973 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.816505 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.817499 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:17.818106 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:17.818232 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:17.821234 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:56213: BlockingRecv error: recv got EOF from 127.10.224.131:56213 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 12:39:18 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:18 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:18.420710 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:18.414734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:47229)
0926 12:39:18.415033 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:18.415052 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:18.415174 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:18.415394 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:18.415396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:18.415658 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:18.415759 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:18.415765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.416548 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.416551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:18.417106 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:18.417114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.417557 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.417560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:18.417651 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:18.418272 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:18.418284 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:18.420577 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
W20260926 12:39:18.420897 20342 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:18.414882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:47229 (local address 127.10.224.190:44809)
0926 12:39:18.415179 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:18.415184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:18.415208 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:39:18.415254 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:18.415256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:18.415300 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:18.415382 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:18.415873 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.416433 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.417213 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.417457 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.417532 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:18.420744 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:47229: BlockingRecv error: recv got EOF from 127.10.224.129:47229 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
Sep 26 12:39:18 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:18 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:18.640381 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:18.634193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:59221)
0926 12:39:18.634549 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:18.634564 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:18.634678 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:18.634986 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:18.634988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:18.635243 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:18.635369 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:18.635378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.636380 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.636384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:18.636955 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:18.636966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.637403 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.637405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:18.637498 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:18.638053 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:18.638064 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:18.640144 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":131,"threads_started":1}
W20260926 12:39:18.640375 20342 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:18.634278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:59221 (local address 127.10.224.190:44809)
0926 12:39:18.634460 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:18.634465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:18.634646 (+ 181us) server_negotiation.cc:407] Connection header received
0926 12:39:18.634803 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:18.634806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:18.634861 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:18.634963 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:18.635561 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.636262 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.637080 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.637298 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.637389 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:18.640272 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:59221: BlockingRecv error: recv got EOF from 127.10.224.130:59221 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 12:39:18 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:18 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:18.828110 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:18.822270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:60793)
0926 12:39:18.822599 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:18.822612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:18.822702 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:18.822991 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:18.822993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:18.823177 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:18.823267 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:18.823272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.824092 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.824095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:18.824644 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:18.824652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.825166 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.825169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:18.825237 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:18.825803 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:18.825815 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:18.827904 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":120,"threads_started":1}
W20260926 12:39:18.828187 20342 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:18.822618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:60793 (local address 127.10.224.190:44809)
0926 12:39:18.822754 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:18.822758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:18.822777 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:39:18.822826 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:18.822828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:18.822871 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:18.822965 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:18.823400 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.823959 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.824776 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:18.825015 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:18.825110 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:18.828070 (+ 2960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:60793: BlockingRecv error: recv got EOF from 127.10.224.131:60793 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:39:19 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:19 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:19.427798 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:19.421785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:58453)
0926 12:39:19.422061 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:19.422074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:19.422169 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:19.422412 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:19.422414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:19.422591 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:19.422685 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:19.422691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.423577 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.423581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:19.424185 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:19.424195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.424678 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.424681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:19.424752 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:19.425449 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:19.425464 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:19.427683 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
W20260926 12:39:19.427960 20346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:19.421846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:58453 (local address 127.10.224.190:44809)
0926 12:39:19.422144 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:19.422148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:19.422162 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:39:19.422243 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:19.422245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:19.422287 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:19.422366 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:19.422823 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.423389 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.424308 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.424546 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.424632 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:19.427821 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:58453: BlockingRecv error: recv got EOF from 127.10.224.129:58453 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":65,"threads_started":1}
Sep 26 12:39:19 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:19 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:19.658524 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:19.641375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:35847)
0926 12:39:19.641635 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:19.641656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:19.641758 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:19.642035 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:19.642037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:19.642218 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:19.642318 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:19.642325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.643112 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.643118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:19.643970 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:19.643983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.644620 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.644623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:19.644709 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:19.655877 (+ 11168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:19.655898 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:19.658369 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
W20260926 12:39:19.658788 20346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:19.641430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:35847 (local address 127.10.224.190:44809)
0926 12:39:19.641584 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:19.641588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:19.641749 (+ 161us) server_negotiation.cc:407] Connection header received
0926 12:39:19.641846 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:19.641848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:19.641909 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:19.641996 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:19.642436 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.642960 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.644093 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.644488 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.644601 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:19.658669 (+ 14068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:35847: BlockingRecv error: recv got EOF from 127.10.224.130:35847 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 12:39:19 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:19 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:19.834550 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:19.828962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:51585)
0926 12:39:19.829314 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:19.829329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:19.829435 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:19.829709 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:19.829711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:19.829895 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:19.830019 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:19.830025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.830768 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.830770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:19.831348 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:19.831356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.831844 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.831847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:19.831912 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:19.832594 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:19.832607 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:19.834418 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
W20260926 12:39:19.834740 20346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:19.829068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:51585 (local address 127.10.224.190:44809)
0926 12:39:19.829222 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:19.829227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:19.829406 (+ 179us) server_negotiation.cc:407] Connection header received
0926 12:39:19.829547 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:19.829549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:19.829598 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:19.829668 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:19.830125 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.830668 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.831451 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:19.831722 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:19.831805 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:19.834578 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:51585: BlockingRecv error: recv got EOF from 127.10.224.131:51585 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 12:39:20 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:20 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:20.434943 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:20.428904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:39187)
0926 12:39:20.429244 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:20.429262 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:20.429391 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:20.429672 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:20.429675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:20.429924 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:20.430043 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:20.430051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.430813 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.430820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:20.431398 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:20.431407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.431917 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.431920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:20.432001 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:20.432552 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:20.432565 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:20.434767 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
W20260926 12:39:20.435082 20350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:20.428932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:39187 (local address 127.10.224.190:44809)
0926 12:39:20.429324 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:20.429328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:20.429347 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:39:20.429503 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:20.429506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:20.429561 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:20.429655 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:20.430164 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.430701 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.431540 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.431784 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.431900 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:20.434921 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:39187: BlockingRecv error: recv got EOF from 127.10.224.129:39187 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
Sep 26 12:39:20 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:20 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:20.665997 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:20.659363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:33453)
0926 12:39:20.659751 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:20.659769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:20.659897 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:20.660195 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:20.660200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:20.660524 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:20.660648 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:20.660657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.661645 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.661648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:20.662267 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:20.662275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.662779 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.662781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:20.662851 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:20.663664 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:20.663677 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:20.665805 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":104,"threads_started":1}
W20260926 12:39:20.666196 20350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:20.659614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:33453 (local address 127.10.224.190:44809)
0926 12:39:20.659763 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:20.659766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:20.659854 (+ 88us) server_negotiation.cc:407] Connection header received
0926 12:39:20.660002 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:20.660007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:20.660071 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:20.660143 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:20.660783 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.661517 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.662388 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.662649 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.662737 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:20.666049 (+ 3312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:33453: BlockingRecv error: recv got EOF from 127.10.224.130:33453 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 12:39:20 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:20 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:20.841490 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:20.835488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:48489)
0926 12:39:20.835829 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:20.835842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:20.835973 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:20.836262 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:20.836265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:20.836513 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:20.836607 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:20.836612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.837397 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.837402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:20.837957 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:20.837965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.838452 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.838455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:20.838539 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:20.839072 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:20.839084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:20.841292 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":82,"threads_started":1}
W20260926 12:39:20.841631 20350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:20.835613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:48489 (local address 127.10.224.190:44809)
0926 12:39:20.835778 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:20.835783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:20.835917 (+ 134us) server_negotiation.cc:407] Connection header received
0926 12:39:20.836086 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:20.836090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:20.836147 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:20.836241 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:20.836745 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.837279 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.838089 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:20.838323 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:20.838438 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:20.841516 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:48489: BlockingRecv error: recv got EOF from 127.10.224.131:48489 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 12:39:21 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:21 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:21.441999 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:21.435949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:34137)
0926 12:39:21.436237 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:21.436250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:21.436347 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:21.436713 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:21.436715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:21.436916 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:21.437010 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:21.437016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.437892 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.437895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:21.438621 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:21.438632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.439106 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.439108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:21.439196 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:21.439837 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:21.439849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:21.441861 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":119,"threads_started":1}
W20260926 12:39:21.442251 20354 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:21.436053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:34137 (local address 127.10.224.190:44809)
0926 12:39:21.436352 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:21.436356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:21.436371 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:39:21.436520 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:21.436523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:21.436592 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:21.436682 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:21.437154 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.437760 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.438763 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.439000 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.439083 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:21.442091 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:34137: BlockingRecv error: recv got EOF from 127.10.224.129:34137 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
Sep 26 12:39:21 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:21 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:21.673590 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:21.666912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:54779)
0926 12:39:21.667269 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:21.667291 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:21.667492 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:21.667798 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:21.667801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:21.668017 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:21.668106 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:21.668112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.668873 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.668879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:21.669551 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:21.669566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.670349 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.670353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:21.670467 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:21.671192 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:21.671205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:21.673394 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":92,"threads_started":1}
W20260926 12:39:21.673728 20354 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:21.667044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:54779 (local address 127.10.224.190:44809)
0926 12:39:21.667236 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:21.667240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:21.667424 (+ 184us) server_negotiation.cc:407] Connection header received
0926 12:39:21.667628 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:21.667631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:21.667685 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:21.667778 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:21.668226 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.668749 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.669726 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.670144 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.670267 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:21.673582 (+ 3315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:54779: BlockingRecv error: recv got EOF from 127.10.224.130:54779 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 12:39:21 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:21 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.131@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:21.849289 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:21.842579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.131:60035)
0926 12:39:21.842937 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:21.842951 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:21.843105 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:21.843492 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:21.843500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:21.843778 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:21.843887 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:21.843894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.844978 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.844985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:21.845676 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:21.845686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.846193 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.846197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:21.846291 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:21.846869 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:21.846882 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:21.849108 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":89,"threads_started":1}
W20260926 12:39:21.849453 20354 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:21.842656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:60035 (local address 127.10.224.190:44809)
0926 12:39:21.842874 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:21.842879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:21.843052 (+ 173us) server_negotiation.cc:407] Connection header received
0926 12:39:21.843234 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:21.843238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:21.843297 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:21.843466 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:21.844052 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.844858 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.845828 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:21.846071 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:21.846189 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:21.849276 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.131:60035: BlockingRecv error: recv got EOF from 127.10.224.131:60035 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 12:39:22 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:22 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.129@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:22.449661 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:22.443156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.129:45021)
0926 12:39:22.443477 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:22.443500 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:22.443678 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:22.444137 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:22.444140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:22.444343 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:22.444444 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:22.444450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.445244 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.445247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:22.445813 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:22.445826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.446332 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.446336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:22.446420 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:22.447173 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:22.447198 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:22.449443 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
W20260926 12:39:22.449774 20358 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:22.443221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:45021 (local address 127.10.224.190:44809)
0926 12:39:22.443481 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:22.443485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:22.443877 (+ 392us) server_negotiation.cc:407] Connection header received
0926 12:39:22.443921 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:22.443924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:22.443974 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:22.444056 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:22.444583 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.445127 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.445957 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.446182 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.446271 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:22.449613 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.129:45021: BlockingRecv error: recv got EOF from 127.10.224.129:45021 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
Sep 26 12:39:22 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:39:22 dist-test-slave-m3pj krb5kdc[19300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.224.130@KRBTEST.COM for oryx/127.10.224.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:39:22.680357 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:22.674675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44809 (local address 127.10.224.130:44015)
0926 12:39:22.674968 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:22.674982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:22.675109 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:22.675350 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:22.675351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:22.675648 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:22.675744 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:22.675750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.676494 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.676496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:22.677082 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:22.677090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.677565 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.677568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:22.677633 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:22.678122 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:22.678133 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:22.680212 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.224.190:44809: Server oryx/127.10.224.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
W20260926 12:39:22.680583 20358 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:22.674760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:44015 (local address 127.10.224.190:44809)
0926 12:39:22.674904 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:22.674909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:22.675083 (+ 174us) server_negotiation.cc:407] Connection header received
0926 12:39:22.675202 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:22.675204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:22.675250 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:22.675318 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:22.675848 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.676371 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.677207 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.677421 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.677538 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:22.680443 (+ 2905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.224.130:44015: BlockingRecv error: recv got EOF from 127.10.224.130:44015 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:39:22.721477 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 19883
I20260926 12:39:22.727243 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 20035
I20260926 12:39:22.733150 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 20167
I20260926 12:39:22.739450 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 19813
2026-09-26T12:39:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18243 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T12:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:39:22Z Disabled control of system clock
I20260926 12:39:22.772115 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:33837
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:33837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 12:39:22.889628 20363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:22.889953 20363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:22.889990 20363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:22.894156 20363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:39:22.894229 20363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:22.894245 20363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:22.894263 20363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:39:22.894277 20363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:39:22.899302 20363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:33837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:33837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:22.900502 20363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:22.901914 20363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:22.911031 20371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:22.911111 20363 server_base.cc:1034] running on GCE node
W20260926 12:39:22.911113 20369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:22.911031 20368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:22.912072 20363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:22.913308 20363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:22.914526 20363 hybrid_clock.cc:648] HybridClock initialized: now 1790426362914496 us; error 53 us; skew 500 ppm
I20260926 12:39:22.916704 20363 webserver.cc:460] Webserver started at http://127.10.224.190:45789/ using document root <none> and password file <none>
I20260926 12:39:22.917433 20363 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:22.917539 20363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:22.917783 20363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:22.920061 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "3601abe4e1474db09219a61736b531e7"
format_stamp: "Formatted at 2026-09-26 12:39:22 on dist-test-slave-m3pj"
server_key: "be0f4d26d78016c12b2a71ba93b9e5dd"
server_key_iv: "1cccd1876fd14795c3041c0f23d9bf05"
server_key_version: "encryptionkey@0"
I20260926 12:39:22.920692 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "3601abe4e1474db09219a61736b531e7"
format_stamp: "Formatted at 2026-09-26 12:39:22 on dist-test-slave-m3pj"
server_key: "be0f4d26d78016c12b2a71ba93b9e5dd"
server_key_iv: "1cccd1876fd14795c3041c0f23d9bf05"
server_key_version: "encryptionkey@0"
I20260926 12:39:22.924837 20363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:39:22.927618 20377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:22.929309 20363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:22.929472 20363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "3601abe4e1474db09219a61736b531e7"
format_stamp: "Formatted at 2026-09-26 12:39:22 on dist-test-slave-m3pj"
server_key: "be0f4d26d78016c12b2a71ba93b9e5dd"
server_key_iv: "1cccd1876fd14795c3041c0f23d9bf05"
server_key_version: "encryptionkey@0"
I20260926 12:39:22.929643 20363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:22.940799 20363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:22.941615 20363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:22.941838 20363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:22.949591 20363 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:33837
I20260926 12:39:22.949640 20429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:33837 every 8 connection(s)
I20260926 12:39:22.950698 20363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:39:22.954314 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:22.959028 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 20363
I20260926 12:39:22.959174 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:39:22.959504 11138 external_mini_cluster.cc:1428] Setting key 9425670cfdaa3ceb01005b90b993cff7
I20260926 12:39:22.961472 20430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7: Bootstrap starting.
I20260926 12:39:22.964178 20430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:22.965030 20430 log.cc:824] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:22.967248 20430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7: No bootstrap required, opened a new log
I20260926 12:39:22.969748 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:22.961161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.10.224.190:33837)
0926 12:39:22.962033 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:22.962053 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:22.962103 (+ 50us) server_negotiation.cc:407] Connection header received
0926 12:39:22.962815 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:22.962833 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:22.963294 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:22.963682 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:22.964154 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.965281 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.966282 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:22.966775 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:22.966974 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:22.967943 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:22.967949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:22.967964 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:39:22.968370 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:22.968426 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:39:22.968453 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:22.968566 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:22.968726 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 12:39:22.969005 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":156,"threads_started":1}
I20260926 12:39:22.971500 20430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601abe4e1474db09219a61736b531e7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33837 } }
I20260926 12:39:22.971828 20430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:22.971909 20430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3601abe4e1474db09219a61736b531e7, State: Initialized, Role: FOLLOWER
I20260926 12:39:22.972414 20430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601abe4e1474db09219a61736b531e7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33837 } }
I20260926 12:39:22.972566 20430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:39:22.972658 20430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:39:22.972764 20430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:22.973826 20430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601abe4e1474db09219a61736b531e7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33837 } }
I20260926 12:39:22.974218 20430 leader_election.cc:302] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3601abe4e1474db09219a61736b531e7; no voters:
I20260926 12:39:22.974573 20430 leader_election.cc:288] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:39:22.974849 20435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:22.975111 20435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3601abe4e1474db09219a61736b531e7, State: Running, Role: LEADER
I20260926 12:39:22.975486 20435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601abe4e1474db09219a61736b531e7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33837 } }
I20260926 12:39:22.976092 20430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:39:22.977242 20437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3601abe4e1474db09219a61736b531e7. Latest consensus state: current_term: 1 leader_uuid: "3601abe4e1474db09219a61736b531e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601abe4e1474db09219a61736b531e7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33837 } } }
I20260926 12:39:22.977375 20437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:22.977699 20436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3601abe4e1474db09219a61736b531e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601abe4e1474db09219a61736b531e7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33837 } } }
I20260926 12:39:22.977856 20436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:22.978199 20444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:39:22.981510 20444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:39:22.988116 20444 catalog_manager.cc:1347] Generated new cluster ID: 776da08c4d834781a2a9fff4445b6a3e
I20260926 12:39:22.988219 20444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:39:23.009202 20444 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:39:23.010263 20444 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:39:23.027124 20444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3601abe4e1474db09219a61736b531e7: Generated new TSK 0
I20260926 12:39:23.028126 20444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:39:23.041805 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33837
--builtin_ntp_servers=127.10.224.148:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 12:39:23.161093 20454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:23.161445 20454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:23.161486 20454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:23.166010 20454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:23.166092 20454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:23.166177 20454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:39:23.171443 20454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:23.172731 20454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:23.174243 20454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:23.185139 20459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:23.185181 20460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:23.185175 20462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:23.185573 20454 server_base.cc:1034] running on GCE node
I20260926 12:39:23.186079 20454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:23.186837 20454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:23.188042 20454 hybrid_clock.cc:648] HybridClock initialized: now 1790426363187999 us; error 53 us; skew 500 ppm
I20260926 12:39:23.190284 20454 webserver.cc:460] Webserver started at http://127.10.224.129:43145/ using document root <none> and password file <none>
I20260926 12:39:23.190920 20454 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:23.190989 20454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:23.191226 20454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:23.193553 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "a827a1fcb1534b00a7be85d0ed66077a"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "bddcf4bf1661021610ea07979c5662cd"
server_key_iv: "4f6ac03170f28860de2a1585560424d8"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.194171 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "a827a1fcb1534b00a7be85d0ed66077a"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "bddcf4bf1661021610ea07979c5662cd"
server_key_iv: "4f6ac03170f28860de2a1585560424d8"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.198526 20454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:39:23.201566 20468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.203153 20454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:23.203323 20454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "a827a1fcb1534b00a7be85d0ed66077a"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "bddcf4bf1661021610ea07979c5662cd"
server_key_iv: "4f6ac03170f28860de2a1585560424d8"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.203627 20454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:23.216665 20454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:23.217599 20454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:23.217792 20454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:23.218401 20454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:23.219489 20454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:23.219599 20454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.219677 20454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:23.219709 20454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.229328 20454 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:43923
I20260926 12:39:23.229393 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:43923 every 8 connection(s)
I20260926 12:39:23.230365 20454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:39:23.239131 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 20454
I20260926 12:39:23.239277 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:39:23.239653 11138 external_mini_cluster.cc:1428] Setting key 97f6de953c4b283c3ac02dbdb67c48e7
I20260926 12:39:23.240912 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:23.232620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:52457 (local address 127.10.224.190:33837)
0926 12:39:23.232825 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:23.232830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:23.234098 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 12:39:23.235687 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:23.235690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:23.235741 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:23.235842 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:23.236821 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.237373 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.238579 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.238858 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.238972 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:23.240409 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:23.240414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:23.240416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:39:23.240430 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:23.240453 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:39:23.240465 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:23.240565 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:23.240738 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 12:39:23.240795 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:39:23.242043 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:23.233132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33837 (local address 127.10.224.129:52457)
0926 12:39:23.233925 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:23.233970 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:23.235399 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:23.236019 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:23.236025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:23.236147 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:23.236660 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:23.236673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.237517 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.237526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:23.238443 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:23.238453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.239038 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.239042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:23.239197 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:23.240004 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:39:23.240026 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:23.240090 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:39:23.240187 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:23.240574 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:23.240584 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:23.240590 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:39:23.240732 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 12:39:23.241062 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":133,"threads_started":1}
I20260926 12:39:23.242174 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33837
--builtin_ntp_servers=127.10.224.148:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 12:39:23.243458 20582 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33837
I20260926 12:39:23.243732 20582 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:23.244323 20582 heartbeater.cc:511] Master 127.10.224.190:33837 requested a full tablet report, sending...
I20260926 12:39:23.245952 20394 ts_manager.cc:194] Registered new tserver with Master: a827a1fcb1534b00a7be85d0ed66077a (127.10.224.129:43923)
I20260926 12:39:23.247161 20394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.129:52457
W20260926 12:39:23.372864 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:23.373246 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:23.373323 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:23.377979 20585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:23.378089 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:23.378180 20585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:39:23.383553 20585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:23.384805 20585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:23.386333 20585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:23.396776 20591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:23.396804 20593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:23.396804 20590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:23.397320 20585 server_base.cc:1034] running on GCE node
I20260926 12:39:23.397758 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:23.398417 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:23.399658 20585 hybrid_clock.cc:648] HybridClock initialized: now 1790426363399636 us; error 63 us; skew 500 ppm
I20260926 12:39:23.401697 20585 webserver.cc:460] Webserver started at http://127.10.224.130:34967/ using document root <none> and password file <none>
I20260926 12:39:23.402379 20585 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:23.402452 20585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:23.402720 20585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:23.404943 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "c6eef9811e1241c684f9dd195cbb8afd"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "cac5c274e529f8335fc7b7a929150cca"
server_key_iv: "da865b679b5e514d2544d3400771167d"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.405550 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "c6eef9811e1241c684f9dd195cbb8afd"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "cac5c274e529f8335fc7b7a929150cca"
server_key_iv: "da865b679b5e514d2544d3400771167d"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.409672 20585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:39:23.412532 20599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.413997 20585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:39:23.414144 20585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "c6eef9811e1241c684f9dd195cbb8afd"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "cac5c274e529f8335fc7b7a929150cca"
server_key_iv: "da865b679b5e514d2544d3400771167d"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.414280 20585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:23.441855 20585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:23.442700 20585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:23.442951 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:23.443702 20585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:23.444908 20585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:23.444969 20585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.445050 20585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:23.445103 20585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.455817 20585 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:41447
I20260926 12:39:23.455837 20712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:41447 every 8 connection(s)
I20260926 12:39:23.456969 20585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:39:23.460327 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 20585
I20260926 12:39:23.460462 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:39:23.460814 11138 external_mini_cluster.cc:1428] Setting key e0efe85ecf03d21975ed9d83033f26e0
I20260926 12:39:23.463117 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33837
--builtin_ntp_servers=127.10.224.148:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 12:39:23.469264 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:23.459081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:34501 (local address 127.10.224.190:33837)
0926 12:39:23.459265 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:23.459271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:23.460790 (+ 1519us) server_negotiation.cc:407] Connection header received
0926 12:39:23.462506 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:23.462509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:23.462564 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:23.462678 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:23.463868 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.464444 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.466477 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.466846 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.466952 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:23.468534 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:23.468538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:23.468540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:39:23.468556 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:23.468576 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:39:23.468584 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:23.468655 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:23.468973 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 12:39:23.469057 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 12:39:23.470688 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:23.459663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33837 (local address 127.10.224.130:34501)
0926 12:39:23.460606 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:23.460658 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:23.462260 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:23.462881 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:23.462889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:23.462988 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:23.463703 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:23.463720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.464633 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.464639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:23.466268 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:23.466285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.467033 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.467041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:23.467313 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:23.468075 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:39:23.468080 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:23.468154 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:39:23.468273 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:23.468688 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:23.468700 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:23.468709 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:39:23.468879 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 12:39:23.469308 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":147,"threads_started":1}
I20260926 12:39:23.472201 20713 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33837
I20260926 12:39:23.472507 20713 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:23.473265 20713 heartbeater.cc:511] Master 127.10.224.190:33837 requested a full tablet report, sending...
I20260926 12:39:23.474700 20394 ts_manager.cc:194] Registered new tserver with Master: c6eef9811e1241c684f9dd195cbb8afd (127.10.224.130:41447)
I20260926 12:39:23.475350 20394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.130:34501
W20260926 12:39:23.586341 20716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:23.586655 20716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:23.586694 20716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:23.591018 20716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:23.591081 20716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:23.591178 20716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:39:23.596573 20716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:23.597752 20716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:23.599200 20716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:23.609402 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:23.609405 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:23.609405 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:23.609483 20716 server_base.cc:1034] running on GCE node
I20260926 12:39:23.610229 20716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:23.610903 20716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:23.612068 20716 hybrid_clock.cc:648] HybridClock initialized: now 1790426363612047 us; error 38 us; skew 500 ppm
I20260926 12:39:23.614068 20716 webserver.cc:460] Webserver started at http://127.10.224.131:39851/ using document root <none> and password file <none>
I20260926 12:39:23.614693 20716 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:23.614768 20716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:23.614971 20716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:23.617295 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "b40a9e146f1a48328fa58ca300235d4b"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "6dd24ac50930648926ab813df764cb04"
server_key_iv: "9a724028fb4395fac04aba91947620bd"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.618196 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "b40a9e146f1a48328fa58ca300235d4b"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "6dd24ac50930648926ab813df764cb04"
server_key_iv: "9a724028fb4395fac04aba91947620bd"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.622943 20716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 12:39:23.625993 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.627411 20716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 12:39:23.627599 20716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "b40a9e146f1a48328fa58ca300235d4b"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "6dd24ac50930648926ab813df764cb04"
server_key_iv: "9a724028fb4395fac04aba91947620bd"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.627735 20716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:23.665199 20716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:23.665979 20716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:23.666186 20716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:23.666814 20716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:23.668016 20716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:23.668069 20716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.668112 20716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:23.668128 20716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.677415 20716 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:43709
I20260926 12:39:23.677497 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:43709 every 8 connection(s)
I20260926 12:39:23.678710 20716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:39:23.681201 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 20716
I20260926 12:39:23.681327 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:39:23.681648 11138 external_mini_cluster.cc:1428] Setting key 47f860ef231a4ea30c81ab17dd4ee12e
I20260926 12:39:23.688939 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:23.680862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:52059 (local address 127.10.224.190:33837)
0926 12:39:23.681000 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:23.681005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:23.682298 (+ 1293us) server_negotiation.cc:407] Connection header received
0926 12:39:23.683847 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:23.683850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:23.683905 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:23.683987 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:23.684949 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.685490 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.686748 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.686992 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.687112 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:23.688402 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:23.688408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:23.688416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:39:23.688430 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:23.688450 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:39:23.688458 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:23.688537 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:23.688706 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 12:39:23.688776 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:39:23.689854 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:23.681283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33837 (local address 127.10.224.131:52059)
0926 12:39:23.682139 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:23.682183 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:23.683640 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:23.684170 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:23.684177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:23.684269 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:23.684808 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:23.684821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.685635 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.685638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:23.686600 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:23.686611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:23.687161 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:23.687165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:23.687351 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:23.688103 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:39:23.688105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:23.688149 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:39:23.688222 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:23.688551 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:23.688560 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:23.688566 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:39:23.688706 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 12:39:23.689026 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":167,"threads_started":1}
I20260926 12:39:23.691102 20844 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33837
I20260926 12:39:23.691388 20844 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:23.692042 20844 heartbeater.cc:511] Master 127.10.224.190:33837 requested a full tablet report, sending...
I20260926 12:39:23.693203 20394 ts_manager.cc:194] Registered new tserver with Master: b40a9e146f1a48328fa58ca300235d4b (127.10.224.131:43709)
I20260926 12:39:23.693908 20394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.131:52059
I20260926 12:39:23.695611 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:39:23.698454 20433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:23.697540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.10.224.190:33837)
0926 12:39:23.697671 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:23.697675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:23.697917 (+ 242us) server_negotiation.cc:407] Connection header received
0926 12:39:23.697978 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:23.697980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:23.698027 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:23.698100 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:23.698340 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53838: BlockingRecv error: recv got EOF from 127.0.0.1:53838 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:39:23.698770 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 20454
I20260926 12:39:23.704876 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 20585
I20260926 12:39:23.710347 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 20716
I20260926 12:39:23.715785 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 20363
2026-09-26T12:39:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (974 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T12:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:39:23Z Disabled control of system clock
I20260926 12:39:23.746654 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:43687
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 12:39:23.864418 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:23.864766 20855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:23.864818 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:23.869398 20855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:39:23.869493 20855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:23.869510 20855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:23.869527 20855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:39:23.869544 20855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:39:23.874985 20855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:43687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:23.876283 20855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:23.877875 20855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:23.887212 20861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:23.887223 20863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:23.887223 20860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:23.887491 20855 server_base.cc:1034] running on GCE node
I20260926 12:39:23.888466 20855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:23.889652 20855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:23.890880 20855 hybrid_clock.cc:648] HybridClock initialized: now 1790426363890854 us; error 40 us; skew 500 ppm
I20260926 12:39:23.893230 20855 webserver.cc:460] Webserver started at http://127.10.224.190:38127/ using document root <none> and password file <none>
I20260926 12:39:23.893888 20855 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:23.893990 20855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:23.894261 20855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:23.896924 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "4d2b70659234468f8bc056c98228b7cb"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "8403ea243d96e5dc1dbe6802b7c15499"
server_key_iv: "d899b44e14a75cae65dcffba5b718bee"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.897572 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "4d2b70659234468f8bc056c98228b7cb"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "8403ea243d96e5dc1dbe6802b7c15499"
server_key_iv: "d899b44e14a75cae65dcffba5b718bee"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.901741 20855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:39:23.904505 20869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:23.905939 20855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:23.906111 20855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "4d2b70659234468f8bc056c98228b7cb"
format_stamp: "Formatted at 2026-09-26 12:39:23 on dist-test-slave-m3pj"
server_key: "8403ea243d96e5dc1dbe6802b7c15499"
server_key_iv: "d899b44e14a75cae65dcffba5b718bee"
server_key_version: "encryptionkey@0"
I20260926 12:39:23.906243 20855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:23.911197 20855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:23.911893 20855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:23.912088 20855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:23.919260 20855 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:43687
I20260926 12:39:23.919281 20921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:43687 every 8 connection(s)
I20260926 12:39:23.920383 20855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:39:23.923417 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 20855
I20260926 12:39:23.923599 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:39:23.923938 11138 external_mini_cluster.cc:1428] Setting key ae29c00e17bccff6379442289deb7eb3
I20260926 12:39:23.924111 20922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:23.929975 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:23.925479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.10.224.190:43687)
0926 12:39:23.926211 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:23.926226 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:23.926263 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:39:23.926981 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:23.926999 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:23.927389 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:23.927811 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:23.928635 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:23.928640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:23.928653 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:39:23.928663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:23.928719 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:39:23.928735 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:23.928894 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:23.928967 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 12:39:23.929219 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":152,"threads_started":1}
I20260926 12:39:23.931656 20922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb: Bootstrap starting.
I20260926 12:39:23.934198 20922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:23.934976 20922 log.cc:824] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:23.937518 20922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb: No bootstrap required, opened a new log
I20260926 12:39:23.941386 20922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2b70659234468f8bc056c98228b7cb" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43687 } }
I20260926 12:39:23.941661 20922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:23.941696 20922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2b70659234468f8bc056c98228b7cb, State: Initialized, Role: FOLLOWER
I20260926 12:39:23.942096 20922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2b70659234468f8bc056c98228b7cb" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43687 } }
I20260926 12:39:23.942225 20922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:39:23.942268 20922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:39:23.942399 20922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:23.943394 20922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2b70659234468f8bc056c98228b7cb" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43687 } }
I20260926 12:39:23.943820 20922 leader_election.cc:302] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d2b70659234468f8bc056c98228b7cb; no voters:
I20260926 12:39:23.944123 20922 leader_election.cc:288] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:39:23.944291 20927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:23.944527 20927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [term 1 LEADER]: Becoming Leader. State: Replica: 4d2b70659234468f8bc056c98228b7cb, State: Running, Role: LEADER
I20260926 12:39:23.944998 20927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2b70659234468f8bc056c98228b7cb" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43687 } }
I20260926 12:39:23.945562 20922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:39:23.946979 20928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d2b70659234468f8bc056c98228b7cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2b70659234468f8bc056c98228b7cb" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43687 } } }
I20260926 12:39:23.947116 20928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:23.947371 20929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d2b70659234468f8bc056c98228b7cb. Latest consensus state: current_term: 1 leader_uuid: "4d2b70659234468f8bc056c98228b7cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2b70659234468f8bc056c98228b7cb" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43687 } } }
I20260926 12:39:23.947588 20929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:23.947585 20938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:39:23.950606 20938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:39:23.957296 20938 catalog_manager.cc:1347] Generated new cluster ID: 2bd7ff07bd3c49d187ec9f2e1c00ef08
I20260926 12:39:23.957397 20938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:39:23.972944 20938 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:39:23.973340 20938 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:39:23.982280 20938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d2b70659234468f8bc056c98228b7cb: Generated new TSK 0
I20260926 12:39:23.983206 20938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:39:23.990617 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43687
--builtin_ntp_servers=127.10.224.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 12:39:24.113828 20946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:24.114142 20946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:24.114187 20946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:24.118616 20946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:24.118692 20946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:24.118755 20946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:39:24.123917 20946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:24.125134 20946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:24.126670 20946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:24.137005 20954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:24.137009 20951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:24.137102 20946 server_base.cc:1034] running on GCE node
W20260926 12:39:24.136999 20952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:24.137982 20946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:24.138764 20946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:24.139962 20946 hybrid_clock.cc:648] HybridClock initialized: now 1790426364139938 us; error 36 us; skew 500 ppm
I20260926 12:39:24.141957 20946 webserver.cc:460] Webserver started at http://127.10.224.129:44417/ using document root <none> and password file <none>
I20260926 12:39:24.142570 20946 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:24.142665 20946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:24.142913 20946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:24.145187 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "dcd7240482424c5aabe6e4d718bd43db"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "6ffd85d319b59583e9e48558c3f1070b"
server_key_iv: "925d569db3d63b3368af608afe28bcfb"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.145821 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "dcd7240482424c5aabe6e4d718bd43db"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "6ffd85d319b59583e9e48558c3f1070b"
server_key_iv: "925d569db3d63b3368af608afe28bcfb"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.150094 20946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:39:24.153210 20960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.154594 20946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:39:24.154752 20946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "dcd7240482424c5aabe6e4d718bd43db"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "6ffd85d319b59583e9e48558c3f1070b"
server_key_iv: "925d569db3d63b3368af608afe28bcfb"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.154891 20946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:24.160044 20946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:24.160727 20946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:24.160936 20946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:24.161521 20946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:24.162504 20946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:24.162575 20946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.162647 20946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:24.162689 20946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.172246 20946 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:46297
I20260926 12:39:24.172262 21073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:46297 every 8 connection(s)
I20260926 12:39:24.173218 20946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:39:24.178023 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 20946
I20260926 12:39:24.178166 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:39:24.178464 11138 external_mini_cluster.cc:1428] Setting key 45d7aff9339fbfa9c3ceaf72e9db2d21
I20260926 12:39:24.180016 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:24.175133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:43379 (local address 127.10.224.190:43687)
0926 12:39:24.175271 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:24.175276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:24.176554 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 12:39:24.178030 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:24.178034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:24.178078 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:24.178217 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:24.179505 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:24.179509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:24.179512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:39:24.179548 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:24.179573 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:39:24.179583 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:24.179688 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:24.179841 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 12:39:24.179885 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:39:24.180568 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43687
--builtin_ntp_servers=127.10.224.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 12:39:24.181010 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:24.175622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43687 (local address 127.10.224.129:43379)
0926 12:39:24.176381 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:24.176425 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:24.177745 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:24.178442 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:24.178451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:24.178575 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:24.179190 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:39:24.179192 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:24.179233 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:39:24.179355 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:24.179685 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:24.179693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:24.179702 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:39:24.179890 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 12:39:24.180166 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260926 12:39:24.182435 21074 heartbeater.cc:347] Connected to a master server at 127.10.224.190:43687
I20260926 12:39:24.182687 21074 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:24.183177 21074 heartbeater.cc:511] Master 127.10.224.190:43687 requested a full tablet report, sending...
I20260926 12:39:24.185082 20886 ts_manager.cc:194] Registered new tserver with Master: dcd7240482424c5aabe6e4d718bd43db (127.10.224.129:46297)
W20260926 12:39:24.299428 21077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:24.299837 21077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:24.299886 21077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:24.304491 21077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:24.304569 21077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:24.304631 21077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:39:24.309976 21077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:24.311301 21077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:24.312849 21077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:24.322847 21082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:24.322860 21083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:24.322868 21085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:24.323153 21077 server_base.cc:1034] running on GCE node
I20260926 12:39:24.323776 21077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:24.324488 21077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:24.325667 21077 hybrid_clock.cc:648] HybridClock initialized: now 1790426364325650 us; error 33 us; skew 500 ppm
I20260926 12:39:24.327867 21077 webserver.cc:460] Webserver started at http://127.10.224.130:44253/ using document root <none> and password file <none>
I20260926 12:39:24.328481 21077 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:24.328547 21077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:24.328768 21077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:24.330992 21077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "4eb4b5884dbd4c8e80238588fcce5513"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "f4ab59915c1d586b23bd7e113a504373"
server_key_iv: "fec55f21c52fd4604e8b792df6993df1"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.331693 21077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "4eb4b5884dbd4c8e80238588fcce5513"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "f4ab59915c1d586b23bd7e113a504373"
server_key_iv: "fec55f21c52fd4604e8b792df6993df1"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.335774 21077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:39:24.338738 21091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.340410 21077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:24.340570 21077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "4eb4b5884dbd4c8e80238588fcce5513"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "f4ab59915c1d586b23bd7e113a504373"
server_key_iv: "fec55f21c52fd4604e8b792df6993df1"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.340710 21077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:24.346042 21077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:24.346881 21077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:24.347082 21077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:24.347885 21077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:24.348929 21077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:24.348977 21077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.349051 21077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:24.349085 21077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.358462 21077 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:38259
I20260926 12:39:24.358489 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:38259 every 8 connection(s)
I20260926 12:39:24.359615 21077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:39:24.367406 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:24.361859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:45923 (local address 127.10.224.190:43687)
0926 12:39:24.361999 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:24.362006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:24.363342 (+ 1336us) server_negotiation.cc:407] Connection header received
0926 12:39:24.365148 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:24.365153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:24.365199 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:24.365295 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:24.366770 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:24.366773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:24.366776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:39:24.366791 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:24.366814 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:39:24.366822 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:24.366924 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:24.367160 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 12:39:24.367218 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:39:24.368049 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 21077
I20260926 12:39:24.368170 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:39:24.368489 11138 external_mini_cluster.cc:1428] Setting key de8173bb763772410997543b107a6959
I20260926 12:39:24.368669 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:24.362331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43687 (local address 127.10.224.130:45923)
0926 12:39:24.363145 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:24.363198 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:24.364766 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:24.365531 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:24.365539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:24.365644 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:24.366425 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:39:24.366428 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:24.366479 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:39:24.366566 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:24.366943 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:24.366952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:24.366958 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:39:24.367152 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 12:39:24.367393 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":131,"threads_started":1}
I20260926 12:39:24.370293 21205 heartbeater.cc:347] Connected to a master server at 127.10.224.190:43687
I20260926 12:39:24.370631 21205 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:24.371116 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43687
--builtin_ntp_servers=127.10.224.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 12:39:24.371138 21205 heartbeater.cc:511] Master 127.10.224.190:43687 requested a full tablet report, sending...
I20260926 12:39:24.372421 20886 ts_manager.cc:194] Registered new tserver with Master: 4eb4b5884dbd4c8e80238588fcce5513 (127.10.224.130:38259)
W20260926 12:39:24.494298 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:24.494626 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:24.494669 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:24.499096 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:24.499176 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:24.499244 21208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:39:24.504609 21208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:24.505874 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:24.507365 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:24.517894 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:24.517894 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:24.517953 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:24.518280 21208 server_base.cc:1034] running on GCE node
I20260926 12:39:24.518759 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:24.519466 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:24.520694 21208 hybrid_clock.cc:648] HybridClock initialized: now 1790426364520659 us; error 49 us; skew 500 ppm
I20260926 12:39:24.522847 21208 webserver.cc:460] Webserver started at http://127.10.224.131:43615/ using document root <none> and password file <none>
I20260926 12:39:24.523478 21208 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:24.523615 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:24.523870 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:24.526049 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "c9c6dec8493848c584b8cf1db9a33c7c"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "492e28057c7f2f8cbf6325efffb8edce"
server_key_iv: "b8508793ab980e316c8a03292af8fede"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.526659 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "c9c6dec8493848c584b8cf1db9a33c7c"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "492e28057c7f2f8cbf6325efffb8edce"
server_key_iv: "b8508793ab980e316c8a03292af8fede"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.530821 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 12:39:24.533905 21222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.535307 21208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:39:24.535571 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "c9c6dec8493848c584b8cf1db9a33c7c"
format_stamp: "Formatted at 2026-09-26 12:39:24 on dist-test-slave-m3pj"
server_key: "492e28057c7f2f8cbf6325efffb8edce"
server_key_iv: "b8508793ab980e316c8a03292af8fede"
server_key_version: "encryptionkey@0"
I20260926 12:39:24.535724 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:24.540784 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:24.541481 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:24.541679 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:24.542249 21208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:24.543205 21208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:24.543277 21208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.543352 21208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:24.543396 21208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:24.553679 21208 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:46195
I20260926 12:39:24.554009 21335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:46195 every 8 connection(s)
I20260926 12:39:24.554802 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:39:24.558691 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 21208
I20260926 12:39:24.558815 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:39:24.559127 11138 external_mini_cluster.cc:1428] Setting key 6304022f565505a695490fc5d592c7e4
I20260926 12:39:24.562415 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:24.557089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:59221 (local address 127.10.224.190:43687)
0926 12:39:24.557231 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:24.557235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:24.558522 (+ 1287us) server_negotiation.cc:407] Connection header received
0926 12:39:24.560384 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:24.560388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:24.560430 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:24.560518 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:24.561953 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:24.561956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:24.561958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:39:24.561971 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:24.561991 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:39:24.561999 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:24.562075 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:24.562261 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 12:39:24.562300 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:39:24.563442 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:24.557519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43687 (local address 127.10.224.131:59221)
0926 12:39:24.558269 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:24.558379 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:24.560139 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:24.560736 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:24.560749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:24.560886 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:24.561588 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:39:24.561591 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:24.561671 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:39:24.561778 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:24.562111 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:24.562120 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:24.562126 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:39:24.562299 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 12:39:24.562594 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":133,"threads_started":1}
I20260926 12:39:24.564842 21336 heartbeater.cc:347] Connected to a master server at 127.10.224.190:43687
I20260926 12:39:24.565099 21336 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:24.565518 21336 heartbeater.cc:511] Master 127.10.224.190:43687 requested a full tablet report, sending...
I20260926 12:39:24.566648 20886 ts_manager.cc:194] Registered new tserver with Master: c9c6dec8493848c584b8cf1db9a33c7c (127.10.224.131:46195)
I20260926 12:39:24.574041 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:39:24.577258 20925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:39:24.576201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.10.224.190:43687)
0926 12:39:24.576342 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:24.576346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:24.576569 (+ 223us) server_negotiation.cc:407] Connection header received
0926 12:39:24.576619 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:24.576621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:24.576656 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:24.576736 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:24.577062 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43804: BlockingRecv error: recv got EOF from 127.0.0.1:43804 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:39:24.577437 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 20946
I20260926 12:39:24.583273 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 21077
I20260926 12:39:24.588908 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 21208
I20260926 12:39:24.594584 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 20855
2026-09-26T12:39:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (878 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:39:24 dist-test-slave-m3pj krb5kdc[21346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:39:24 dist-test-slave-m3pj krb5kdc[21346](info): set up 2 sockets
Sep 26 12:39:24 dist-test-slave-m3pj krb5kdc[21346](info): commencing operation
krb5kdc: starting...
W20260926 12:39:25.657807 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:39:25 dist-test-slave-m3pj krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:39:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:39:25.799798 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:44201
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:25.917312 21359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:25.917675 21359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:25.917769 21359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:25.922264 21359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:39:25.922416 21359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:25.922454 21359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:25.922494 21359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:39:25.922533 21359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:39:25.928699 21359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:25.930037 21359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:25.931239 21359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:25.941214 21364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:25.941190 21365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:25.941341 21359 server_base.cc:1034] running on GCE node
W20260926 12:39:25.941193 21367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:25.942160 21359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:25.943408 21359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:25.944597 21359 hybrid_clock.cc:648] HybridClock initialized: now 1790426365944581 us; error 36 us; skew 500 ppm
Sep 26 12:39:25 dist-test-slave-m3pj krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426365, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:25.947877 21359 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:39:25.948933 21359 webserver.cc:460] Webserver started at http://127.10.224.190:34823/ using document root <none> and password file <none>
I20260926 12:39:25.949585 21359 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:25.949694 21359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:25.949942 21359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:25.952293 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "ebc7305616444a2a80f7d1b59341d04d"
format_stamp: "Formatted at 2026-09-26 12:39:25 on dist-test-slave-m3pj"
server_key: "4f1ef5ecc35d0ce55af9f3a66b0aff24"
server_key_iv: "384fd98f5e37d13add8a85798632f14f"
server_key_version: "encryptionkey@0"
I20260926 12:39:25.952931 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "ebc7305616444a2a80f7d1b59341d04d"
format_stamp: "Formatted at 2026-09-26 12:39:25 on dist-test-slave-m3pj"
server_key: "4f1ef5ecc35d0ce55af9f3a66b0aff24"
server_key_iv: "384fd98f5e37d13add8a85798632f14f"
server_key_version: "encryptionkey@0"
I20260926 12:39:25.957268 21359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:39:25.960446 21374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:25.962091 21359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:25.962378 21359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "ebc7305616444a2a80f7d1b59341d04d"
format_stamp: "Formatted at 2026-09-26 12:39:25 on dist-test-slave-m3pj"
server_key: "4f1ef5ecc35d0ce55af9f3a66b0aff24"
server_key_iv: "384fd98f5e37d13add8a85798632f14f"
server_key_version: "encryptionkey@0"
I20260926 12:39:25.962589 21359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:25.993072 21359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:25.997615 21359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:25.997889 21359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:26.006135 21359 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:44201
I20260926 12:39:26.006453 21426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:44201 every 8 connection(s)
I20260926 12:39:26.007406 21359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:39:26.011368 21427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:26.017560 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 21359
I20260926 12:39:26.017679 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:39:26.017792 21427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d: Bootstrap starting.
I20260926 12:39:26.018045 11138 external_mini_cluster.cc:1428] Setting key 6534dfc6e97726cf70d3d98c4120d50e
I20260926 12:39:26.020849 21427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:26.021838 21427 log.cc:824] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:26.024739 21427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d: No bootstrap required, opened a new log
Sep 26 12:39:26 dist-test-slave-m3pj krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:39:26.029768 21427 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc7305616444a2a80f7d1b59341d04d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44201 } }
I20260926 12:39:26.030076 21427 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:26.030160 21427 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc7305616444a2a80f7d1b59341d04d, State: Initialized, Role: FOLLOWER
I20260926 12:39:26.030812 21427 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc7305616444a2a80f7d1b59341d04d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44201 } }
I20260926 12:39:26.031039 21427 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:39:26.031139 21427 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:39:26.031277 21427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:26.032419 21427 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc7305616444a2a80f7d1b59341d04d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44201 } }
I20260926 12:39:26.032831 21427 leader_election.cc:302] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebc7305616444a2a80f7d1b59341d04d; no voters:
I20260926 12:39:26.033226 21427 leader_election.cc:288] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:39:26.033309 21432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:26.033519 21432 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [term 1 LEADER]: Becoming Leader. State: Replica: ebc7305616444a2a80f7d1b59341d04d, State: Running, Role: LEADER
I20260926 12:39:26.033871 21432 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc7305616444a2a80f7d1b59341d04d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44201 } }
I20260926 12:39:26.034721 21427 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:39:26.035815 21434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebc7305616444a2a80f7d1b59341d04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc7305616444a2a80f7d1b59341d04d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44201 } } }
I20260926 12:39:26.035943 21434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:26.036252 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc7305616444a2a80f7d1b59341d04d. Latest consensus state: current_term: 1 leader_uuid: "ebc7305616444a2a80f7d1b59341d04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc7305616444a2a80f7d1b59341d04d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44201 } } }
I20260926 12:39:26.036363 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d [sys.catalog]: This master's current role is: LEADER
I20260926 12:39:26.036489 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.019672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.10.224.190:44201)
0926 12:39:26.020153 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.020173 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.020228 (+ 55us) server_negotiation.cc:407] Connection header received
0926 12:39:26.021198 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.021225 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.021721 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.022113 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:26.023084 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.024720 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.026066 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.026600 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.026743 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.030028 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:26.030034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:26.030051 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:26.030116 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:26.032383 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.032947 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.032954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.032963 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.033124 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.033473 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.033477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.033479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.034192 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:26.034433 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.034787 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.035243 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":153,"threads_started":1}
I20260926 12:39:26.036664 21441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:39:26.040796 21441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:39:26.048004 21441 catalog_manager.cc:1347] Generated new cluster ID: 19fc11d8763a4043835b6acce1742124
I20260926 12:39:26.048103 21441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:39:26.075928 21441 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:39:26.077318 21441 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:39:26.084399 21441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d: Generated new TSK 0
I20260926 12:39:26.085345 21441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:39:26.120754 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44201
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:26.245458 21453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:26.245792 21453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:26.245929 21453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:26.250479 21453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:26.250559 21453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:26.250625 21453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:39:26.257213 21453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:26.258450 21453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:26.259714 21453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:26.270710 21459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:26.270699 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:26.270694 21461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:26.271073 21453 server_base.cc:1034] running on GCE node
I20260926 12:39:26.271720 21453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:26.272500 21453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:26.273706 21453 hybrid_clock.cc:648] HybridClock initialized: now 1790426366273688 us; error 47 us; skew 500 ppm
Sep 26 12:39:26 dist-test-slave-m3pj krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:26.276866 21453 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:39:26.277947 21453 webserver.cc:460] Webserver started at http://127.10.224.129:40051/ using document root <none> and password file <none>
I20260926 12:39:26.278602 21453 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:26.278707 21453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:26.278997 21453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:26.281311 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "d1dde06831604c0c963d109f6522165c"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "09301087fc4df6a48fa4f27b9182142c"
server_key_iv: "cd59fa17a575f72ac917d5ef552d9c87"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.281945 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "d1dde06831604c0c963d109f6522165c"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "09301087fc4df6a48fa4f27b9182142c"
server_key_iv: "cd59fa17a575f72ac917d5ef552d9c87"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.286068 21453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:39:26.288986 21468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.290917 21453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:39:26.291102 21453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "d1dde06831604c0c963d109f6522165c"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "09301087fc4df6a48fa4f27b9182142c"
server_key_iv: "cd59fa17a575f72ac917d5ef552d9c87"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.291247 21453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:26.314739 21453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:26.318472 21453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:26.318723 21453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:26.319403 21453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:26.320369 21453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:26.320446 21453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.320521 21453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:26.320568 21453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.330929 21453 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:43491
I20260926 12:39:26.330953 21581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:43491 every 8 connection(s)
I20260926 12:39:26.331955 21453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:39:26.338604 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 21453
I20260926 12:39:26.338719 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:39:26.339004 11138 external_mini_cluster.cc:1428] Setting key 231a3aadd667dc8ea58ed851bba83e06
Sep 26 12:39:26 dist-test-slave-m3pj krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:39:26.347939 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.334008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:38891 (local address 127.10.224.190:44201)
0926 12:39:26.334165 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.334171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.335280 (+ 1109us) server_negotiation.cc:407] Connection header received
0926 12:39:26.336898 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.336901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.336960 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.337056 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:26.338441 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.339078 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.340635 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.340857 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.340964 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.343983 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:26.343987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:26.343989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:26.344015 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:26.345590 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.346190 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.346198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.346199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.346278 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.346726 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.346730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.346731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.346937 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:26.347045 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.347624 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.347762 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:39:26.348923 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.334606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44201 (local address 127.10.224.129:38891)
0926 12:39:26.335112 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.335159 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:26.336632 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:26.337268 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:26.337280 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:26.337706 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:26.338249 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.338264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.339217 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.339222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.340513 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.340527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.341032 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.341044 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.341300 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.341985 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:26.342013 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:26.343773 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:26.345743 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:26.345750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:26.345770 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:26.346069 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:26.346456 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:26.346459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:26.346461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:26.346619 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:26.347066 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:26.347073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:26.347357 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 12:39:26.347581 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 12:39:26.347965 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260926 12:39:26.350548 21582 heartbeater.cc:347] Connected to a master server at 127.10.224.190:44201
I20260926 12:39:26.350835 21582 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:26.351764 21582 heartbeater.cc:511] Master 127.10.224.190:44201 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:39:26.353749 21391 ts_manager.cc:194] Registered new tserver with Master: d1dde06831604c0c963d109f6522165c (127.10.224.129:43491)
I20260926 12:39:26.355094 21391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:38891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:39:26.371717 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44201
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:26.492417 21587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:26.492777 21587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:26.492861 21587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:26.497455 21587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:26.497571 21587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:26.497675 21587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:39:26.503693 21587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:26.504958 21587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:26.506090 21587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:26.516860 21595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:26.516870 21593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:26.516863 21592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:26.517138 21587 server_base.cc:1034] running on GCE node
I20260926 12:39:26.517632 21587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:26.518314 21587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:26.519572 21587 hybrid_clock.cc:648] HybridClock initialized: now 1790426366519486 us; error 51 us; skew 500 ppm
Sep 26 12:39:26 dist-test-slave-m3pj krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:26.522754 21587 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:39:26.523891 21587 webserver.cc:460] Webserver started at http://127.10.224.130:40223/ using document root <none> and password file <none>
I20260926 12:39:26.524513 21587 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:26.524578 21587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:26.524807 21587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:26.527048 21587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "3a5b19c642cb4719aba25dcec943ad1b"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "bcfb48ad9d83a7877453a6b41f7878a0"
server_key_iv: "2c89c569584e05bd7534e560bab9086b"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.527696 21587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "3a5b19c642cb4719aba25dcec943ad1b"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "bcfb48ad9d83a7877453a6b41f7878a0"
server_key_iv: "2c89c569584e05bd7534e560bab9086b"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.531853 21587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:39:26.534518 21602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.536371 21587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:26.536551 21587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "3a5b19c642cb4719aba25dcec943ad1b"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "bcfb48ad9d83a7877453a6b41f7878a0"
server_key_iv: "2c89c569584e05bd7534e560bab9086b"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.536684 21587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:26.555033 21587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:26.558780 21587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:26.559031 21587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:26.559751 21587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:26.560755 21587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:26.560834 21587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.560904 21587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:26.560956 21587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.571310 21587 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:35991
I20260926 12:39:26.571389 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:35991 every 8 connection(s)
I20260926 12:39:26.572458 21587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:39:26.579319 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 21587
I20260926 12:39:26.579567 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:39:26.579864 11138 external_mini_cluster.cc:1428] Setting key 96d16287b7a98dad5e798c9e3552528a
Sep 26 12:39:26 dist-test-slave-m3pj krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:39:26.590327 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.574620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:57433 (local address 127.10.224.190:44201)
0926 12:39:26.574792 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.574797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.575957 (+ 1160us) server_negotiation.cc:407] Connection header received
0926 12:39:26.577517 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.577520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.577575 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.577665 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:26.579278 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.579984 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.581578 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.581925 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.582055 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.585512 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:26.585519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:26.585526 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:26.585566 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:26.587317 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.588236 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.588239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.588240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.588330 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.588852 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.588857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.588858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.589079 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:26.589160 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.589984 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.590138 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:39:26.591575 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.575029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44201 (local address 127.10.224.130:57433)
0926 12:39:26.575757 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.575827 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:26.577208 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:26.577924 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:26.577932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:26.578433 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:26.579056 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.579073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.580151 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.580154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.581301 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.581314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.582146 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.582150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.582404 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.583138 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:26.583156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:26.585281 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:26.587481 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:26.587495 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:26.587578 (+ 83us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:26.588097 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:26.588492 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:26.588496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:26.588500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:26.588727 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:26.589197 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:26.589208 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:26.589645 (+ 437us) client_negotiation.cc:769] Sending connection context
0926 12:39:26.589892 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 12:39:26.590317 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":141,"threads_started":1}
I20260926 12:39:26.593014 21716 heartbeater.cc:347] Connected to a master server at 127.10.224.190:44201
I20260926 12:39:26.593425 21716 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:39:26.594166 21716 heartbeater.cc:511] Master 127.10.224.190:44201 requested a full tablet report, sending...
I20260926 12:39:26.595588 21391 ts_manager.cc:194] Registered new tserver with Master: 3a5b19c642cb4719aba25dcec943ad1b (127.10.224.130:35991)
I20260926 12:39:26.596560 21391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:57433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:39:26.611799 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44201
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:39:26.732668 21721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:39:26.732975 21721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:39:26.733022 21721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:39:26.737453 21721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:39:26.737534 21721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:39:26.737597 21721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:39:26.743471 21721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:39:26.744681 21721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:39:26.745720 21721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:39:26.756513 21729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:26.756551 21726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:39:26.756536 21727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:39:26.756780 21721 server_base.cc:1034] running on GCE node
I20260926 12:39:26.757364 21721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:39:26.758103 21721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:39:26.759320 21721 hybrid_clock.cc:648] HybridClock initialized: now 1790426366759299 us; error 39 us; skew 500 ppm
Sep 26 12:39:26 dist-test-slave-m3pj krb5kdc[21346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:39:26.762635 21721 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:39:26.763736 21721 webserver.cc:460] Webserver started at http://127.10.224.131:33291/ using document root <none> and password file <none>
I20260926 12:39:26.764494 21721 fs_manager.cc:359] Metadata directory not provided
I20260926 12:39:26.764611 21721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:39:26.764868 21721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:39:26.767066 21721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "95d8270f836f4916bae1e3ef6bfb17cc"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "dce8581fdea414875299dc48044ce4bd"
server_key_iv: "f8ffac107ea833c335f262ab33661374"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.767728 21721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "95d8270f836f4916bae1e3ef6bfb17cc"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "dce8581fdea414875299dc48044ce4bd"
server_key_iv: "f8ffac107ea833c335f262ab33661374"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.771859 21721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:39:26.774535 21736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.775983 21721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:39:26.776134 21721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "95d8270f836f4916bae1e3ef6bfb17cc"
format_stamp: "Formatted at 2026-09-26 12:39:26 on dist-test-slave-m3pj"
server_key: "dce8581fdea414875299dc48044ce4bd"
server_key_iv: "f8ffac107ea833c335f262ab33661374"
server_key_version: "encryptionkey@0"
I20260926 12:39:26.776304 21721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:39:26.794519 21721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:39:26.798281 21721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:39:26.798533 21721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:39:26.799176 21721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:39:26.800119 21721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:39:26.800194 21721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.800269 21721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:39:26.800311 21721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:39:26.810125 21721 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:44507
I20260926 12:39:26.810142 21849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:44507 every 8 connection(s)
I20260926 12:39:26.811072 21721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:39:26.819310 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 21721
I20260926 12:39:26.819550 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:39:26.819888 11138 external_mini_cluster.cc:1428] Setting key f6c27235f48e3ead78b3f6622e66ce97
Sep 26 12:39:26 dist-test-slave-m3pj krb5kdc[21346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426366, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:39:26.825909 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.812963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:43781 (local address 127.10.224.190:44201)
0926 12:39:26.813114 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.813117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.813930 (+ 813us) server_negotiation.cc:407] Connection header received
0926 12:39:26.815354 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.815358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.815421 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.815532 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:26.816817 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.817355 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.818543 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.818781 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.818906 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.821888 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:26.821892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:26.821895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:26.821932 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:26.823622 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.824205 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.824210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.824212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.824306 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.824701 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.824705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.824707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.824936 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:26.825049 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.825601 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.825737 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:39:26.826844 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.813300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:44201 (local address 127.10.224.131:43781)
0926 12:39:26.813762 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.813807 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:26.815114 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:26.815743 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:26.815750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:26.816130 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:39:26.816624 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.816639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.817490 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.817494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.818409 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.818418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.818953 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.818957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.819140 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.819885 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:39:26.819903 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:39:26.821697 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:39:26.823766 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:26.823774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:26.823794 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:26.824089 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:26.824411 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:39:26.824416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:39:26.824418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:39:26.824591 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:39:26.825050 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:39:26.825060 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:39:26.825350 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 12:39:26.825564 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 12:39:26.825937 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260926 12:39:26.828179 21850 heartbeater.cc:347] Connected to a master server at 127.10.224.190:44201
I20260926 12:39:26.828441 21850 heartbeater.cc:464] Registering TS with master...
I20260926 12:39:26.828951 21850 heartbeater.cc:511] Master 127.10.224.190:44201 requested a full tablet report, sending...
I20260926 12:39:26.830037 21391 ts_manager.cc:194] Registered new tserver with Master: 95d8270f836f4916bae1e3ef6bfb17cc (127.10.224.131:44507)
I20260926 12:39:26.830579 21391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:43781
I20260926 12:39:26.834077 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:39:26.844146 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.836311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.10.224.190:44201)
0926 12:39:26.836477 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.836480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.836634 (+ 154us) server_negotiation.cc:407] Connection header received
0926 12:39:26.836724 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.836726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.836764 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.836853 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:39:26.837679 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.838203 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.839095 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.839305 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.839406 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.840912 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:39:26.840914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:39:26.840916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:39:26.840941 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:39:26.842406 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.842842 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.842845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.842847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.842923 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:39:26.843211 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:39:26.843214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:39:26.843215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:39:26.843481 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:39:26.843673 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.843886 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.844009 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 12:39:26.847291 21391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:39:26.850512 21391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:39:26.866307 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.861468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.10.224.131:44507)
0926 12:39:26.861855 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.861868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.861891 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:39:26.861966 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.861974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.862147 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.862313 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:26.862527 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.863422 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.865004 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.865583 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.865727 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.865889 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.866057 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.866158 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":90,"threads_started":1}
I20260926 12:39:26.866304 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.860262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:44507 (local address 127.0.0.1:33670)
0926 12:39:26.861332 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.861348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:26.861446 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:26.862286 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:26.862289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:26.862306 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:26.862402 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.862409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.863584 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.863587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.864880 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.864890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.865713 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.865716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.865825 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.865861 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 12:39:26.865984 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 12:39:26.866121 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":58,"thread_start_us":131,"threads_started":1}
I20260926 12:39:26.867491 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.861043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:35991 (local address 127.0.0.1:52314)
0926 12:39:26.861794 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.861807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:26.861921 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:26.862473 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:26.862477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:26.862494 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:26.862572 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.862577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.864157 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.864160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.865629 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.865641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.867048 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.867052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.867142 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.867165 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:39:26.867241 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 12:39:26.867326 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":106,"thread_start_us":121,"threads_started":1}
I20260926 12:39:26.867556 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.860181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:43491 (local address 127.0.0.1:37780)
0926 12:39:26.860741 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.860809 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:26.860959 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:26.862058 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:26.862064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:26.862090 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:26.862274 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.862285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.864079 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.864085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.865630 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.865641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.867103 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.867107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.867200 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.867225 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:39:26.867302 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:39:26.867379 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":81,"thread_start_us":96,"threads_started":1}
I20260926 12:39:26.867965 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.861402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.10.224.130:35991)
0926 12:39:26.861720 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.861731 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.861917 (+ 186us) server_negotiation.cc:407] Connection header received
0926 12:39:26.862031 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.862037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.862233 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.862398 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:26.862689 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.864004 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.865788 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.866690 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.866823 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.866974 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.867691 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.867801 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":121,"threads_started":1}
I20260926 12:39:26.868779 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.860945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.10.224.129:43491)
0926 12:39:26.861456 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.861465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.861494 (+ 29us) server_negotiation.cc:407] Connection header received
0926 12:39:26.861586 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.861593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.861843 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.862445 (+ 602us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:26.862584 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.863908 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.865788 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.866962 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.867091 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.868542 (+ 1451us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.868587 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.868660 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":155,"threads_started":1}
I20260926 12:39:26.870062 21766 tablet_service.cc:1467] Processing CreateTablet for tablet 4caff133fff147c1b2fbb1856d70f910 (DEFAULT_TABLE table=test-table [id=dc9818ddafe94256b15e741e77bb4376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:26.871358 21766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caff133fff147c1b2fbb1856d70f910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:26.870975 21650 tablet_service.cc:1467] Processing CreateTablet for tablet 4caff133fff147c1b2fbb1856d70f910 (DEFAULT_TABLE table=test-table [id=dc9818ddafe94256b15e741e77bb4376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:26.871094 21516 tablet_service.cc:1467] Processing CreateTablet for tablet 4caff133fff147c1b2fbb1856d70f910 (DEFAULT_TABLE table=test-table [id=dc9818ddafe94256b15e741e77bb4376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:39:26.872062 21650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caff133fff147c1b2fbb1856d70f910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:26.872118 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4caff133fff147c1b2fbb1856d70f910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:39:26.878539 21867 tablet_bootstrap.cc:492] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c: Bootstrap starting.
I20260926 12:39:26.879827 21868 tablet_bootstrap.cc:492] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: Bootstrap starting.
I20260926 12:39:26.880234 21869 tablet_bootstrap.cc:492] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: Bootstrap starting.
I20260926 12:39:26.881501 21867 tablet_bootstrap.cc:654] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:26.882072 21868 tablet_bootstrap.cc:654] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:26.882302 21867 log.cc:824] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:26.882855 21869 tablet_bootstrap.cc:654] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: Neither blocks nor log segments found. Creating new log.
I20260926 12:39:26.883400 21868 log.cc:824] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:26.883661 21869 log.cc:824] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:39:26.885118 21867 tablet_bootstrap.cc:492] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c: No bootstrap required, opened a new log
I20260926 12:39:26.885306 21868 tablet_bootstrap.cc:492] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: No bootstrap required, opened a new log
I20260926 12:39:26.885372 21867 ts_tablet_manager.cc:1397] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 12:39:26.885521 21868 ts_tablet_manager.cc:1397] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 12:39:26.885624 21869 tablet_bootstrap.cc:492] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: No bootstrap required, opened a new log
I20260926 12:39:26.885809 21869 ts_tablet_manager.cc:1397] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:39:26.888543 21868 raft_consensus.cc:348] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.888799 21868 raft_consensus.cc:374] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:26.888859 21868 raft_consensus.cc:729] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d8270f836f4916bae1e3ef6bfb17cc, State: Initialized, Role: FOLLOWER
I20260926 12:39:26.889325 21868 consensus_queue.cc:260] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.889674 21869 raft_consensus.cc:348] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.889951 21869 raft_consensus.cc:374] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:26.890012 21869 raft_consensus.cc:729] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5b19c642cb4719aba25dcec943ad1b, State: Initialized, Role: FOLLOWER
I20260926 12:39:26.890097 21850 heartbeater.cc:503] Master 127.10.224.190:44201 was elected leader, sending a full tablet report...
I20260926 12:39:26.890432 21868 ts_tablet_manager.cc:1428] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260926 12:39:26.890439 21869 consensus_queue.cc:260] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.891134 21716 heartbeater.cc:503] Master 127.10.224.190:44201 was elected leader, sending a full tablet report...
I20260926 12:39:26.891474 21869 ts_tablet_manager.cc:1428] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:39:26.891635 21867 raft_consensus.cc:348] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.891902 21867 raft_consensus.cc:374] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:39:26.891959 21867 raft_consensus.cc:729] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1dde06831604c0c963d109f6522165c, State: Initialized, Role: FOLLOWER
I20260926 12:39:26.892377 21867 consensus_queue.cc:260] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.893085 21582 heartbeater.cc:503] Master 127.10.224.190:44201 was elected leader, sending a full tablet report...
I20260926 12:39:26.893433 21867 ts_tablet_manager.cc:1428] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 12:39:26.898949 21875 raft_consensus.cc:482] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:39:26.899164 21875 raft_consensus.cc:504] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.900409 21875 leader_election.cc:288] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95d8270f836f4916bae1e3ef6bfb17cc (127.10.224.131:44507), 3a5b19c642cb4719aba25dcec943ad1b (127.10.224.130:35991)
I20260926 12:39:26.904515 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.900888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:37291 (local address 127.10.224.130:35991)
0926 12:39:26.901023 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.901028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.901380 (+ 352us) server_negotiation.cc:407] Connection header received
0926 12:39:26.901532 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.901535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.901587 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.901665 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:26.901932 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.902413 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.903464 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.903944 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.904076 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.904133 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.904306 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.904376 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:39:26.904542 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.900773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:35991 (local address 127.10.224.129:37291)
0926 12:39:26.901275 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.901286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:26.901421 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:26.901687 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:26.901689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:26.901705 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:26.901825 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.901830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.902519 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.902525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.903330 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.903339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.904070 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.904073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.904170 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.904193 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:39:26.904292 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 12:39:26.904376 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260926 12:39:26.904599 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.901121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:37975 (local address 127.10.224.131:44507)
0926 12:39:26.901249 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.901252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.901275 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:39:26.901320 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.901322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.901360 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.901458 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:39:26.901731 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.902236 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.903543 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.903988 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.904125 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.904181 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.904398 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.904474 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:39:26.904655 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.900589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:44507 (local address 127.10.224.129:37975)
0926 12:39:26.900921 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.900938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:39:26.901055 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:39:26.901463 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:39:26.901466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:39:26.901484 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:39:26.901622 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.901627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.902353 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.902357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.903379 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:39:26.903388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.904125 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.904129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:39:26.904238 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.904259 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:39:26.904362 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 12:39:26.904454 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":101,"thread_start_us":163,"threads_started":1}
I20260926 12:39:26.905253 21804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caff133fff147c1b2fbb1856d70f910" candidate_uuid: "d1dde06831604c0c963d109f6522165c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" is_pre_election: true
I20260926 12:39:26.905493 21670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caff133fff147c1b2fbb1856d70f910" candidate_uuid: "d1dde06831604c0c963d109f6522165c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5b19c642cb4719aba25dcec943ad1b" is_pre_election: true
I20260926 12:39:26.905617 21804 raft_consensus.cc:2454] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1dde06831604c0c963d109f6522165c in term 0.
I20260926 12:39:26.905810 21670 raft_consensus.cc:2454] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1dde06831604c0c963d109f6522165c in term 0.
I20260926 12:39:26.906139 21470 leader_election.cc:302] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d8270f836f4916bae1e3ef6bfb17cc, d1dde06831604c0c963d109f6522165c; no voters:
I20260926 12:39:26.906472 21875 raft_consensus.cc:2781] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:39:26.906558 21875 raft_consensus.cc:482] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:39:26.906601 21875 raft_consensus.cc:3037] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:26.907730 21875 raft_consensus.cc:504] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.908094 21875 leader_election.cc:288] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [CANDIDATE]: Term 1 election: Requested vote from peers 95d8270f836f4916bae1e3ef6bfb17cc (127.10.224.131:44507), 3a5b19c642cb4719aba25dcec943ad1b (127.10.224.130:35991)
I20260926 12:39:26.908414 21804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caff133fff147c1b2fbb1856d70f910" candidate_uuid: "d1dde06831604c0c963d109f6522165c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d8270f836f4916bae1e3ef6bfb17cc"
I20260926 12:39:26.908463 21670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4caff133fff147c1b2fbb1856d70f910" candidate_uuid: "d1dde06831604c0c963d109f6522165c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5b19c642cb4719aba25dcec943ad1b"
I20260926 12:39:26.908555 21804 raft_consensus.cc:3037] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:26.908628 21670 raft_consensus.cc:3037] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:39:26.909863 21804 raft_consensus.cc:2454] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1dde06831604c0c963d109f6522165c in term 1.
I20260926 12:39:26.910009 21670 raft_consensus.cc:2454] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1dde06831604c0c963d109f6522165c in term 1.
I20260926 12:39:26.910135 21470 leader_election.cc:302] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d8270f836f4916bae1e3ef6bfb17cc, d1dde06831604c0c963d109f6522165c; no voters:
I20260926 12:39:26.910321 21875 raft_consensus.cc:2781] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:39:26.910534 21875 raft_consensus.cc:686] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [term 1 LEADER]: Becoming Leader. State: Replica: d1dde06831604c0c963d109f6522165c, State: Running, Role: LEADER
I20260926 12:39:26.910871 21875 consensus_queue.cc:237] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } }
I20260926 12:39:26.914265 21390 catalog_manager.cc:5495] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c reported cstate change: term changed from 0 to 1, leader changed from <none> to d1dde06831604c0c963d109f6522165c (127.10.224.129). New cstate: current_term: 1 leader_uuid: "d1dde06831604c0c963d109f6522165c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1dde06831604c0c963d109f6522165c" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 } health_report { overall_health: UNKNOWN } } }
I20260926 12:39:26.931345 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.927477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.10.224.129:43491)
0926 12:39:26.927658 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 12:39:26.927662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:39:26.927677 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:39:26.927850 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:39:26.927852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:39:26.927913 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:39:26.928045 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:39:26.928289 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.928829 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.929728 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:39:26.929963 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:39:26.930107 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:39:26.930488 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:39:26.930945 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:39:26.931083 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 12:39:26.931124 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 12:39:26.931186 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 12:39:26.938545 21804 raft_consensus.cc:1261] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 1 FOLLOWER]: Refusing update from remote peer d1dde06831604c0c963d109f6522165c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:26.938722 21670 raft_consensus.cc:1261] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 1 FOLLOWER]: Refusing update from remote peer d1dde06831604c0c963d109f6522165c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:39:26.939458 21878 consensus_queue.cc:1035] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d8270f836f4916bae1e3ef6bfb17cc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:26.939740 21875 consensus_queue.cc:1035] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5b19c642cb4719aba25dcec943ad1b" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:39:26.961508 21391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:26.961699 21391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:39:26.964932 21391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebc7305616444a2a80f7d1b59341d04d: Sending DeleteTablet for 3 replicas of tablet 4caff133fff147c1b2fbb1856d70f910
I20260926 12:39:26.965986 21516 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caff133fff147c1b2fbb1856d70f910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:26 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:37780
I20260926 12:39:26.965987 21650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caff133fff147c1b2fbb1856d70f910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:26 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:52314
I20260926 12:39:26.966511 21766 tablet_service.cc:1514] Processing DeleteTablet for tablet 4caff133fff147c1b2fbb1856d70f910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:39:26 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:33670
I20260926 12:39:26.966569 21891 tablet_replica.cc:330] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: stopping tablet replica
I20260926 12:39:26.966580 21892 tablet_replica.cc:330] T 4caff133fff147c1b2fbb1856d70f910 P d1dde06831604c0c963d109f6522165c: stopping tablet replica
I20260926 12:39:26.966905 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 21453
I20260926 12:39:26.967566 21891 raft_consensus.cc:2229] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:26.968070 21891 raft_consensus.cc:2258] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:26.968796 21893 tablet_replica.cc:330] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: stopping tablet replica
I20260926 12:39:26.969187 21893 raft_consensus.cc:2229] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:39:26.969579 21893 raft_consensus.cc:2258] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:39:26.970088 21891 ts_tablet_manager.cc:1905] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:26.971149 21893 ts_tablet_manager.cc:1905] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:39:26.973670 21891 ts_tablet_manager.cc:1918] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:39:26.973805 21891 log.cc:1196] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/4caff133fff147c1b2fbb1856d70f910
I20260926 12:39:26.974205 21891 ts_tablet_manager.cc:1939] T 4caff133fff147c1b2fbb1856d70f910 P 3a5b19c642cb4719aba25dcec943ad1b: Deleting consensus metadata
I20260926 12:39:26.975015 21893 ts_tablet_manager.cc:1918] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:39:26.975121 21893 log.cc:1196] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/4caff133fff147c1b2fbb1856d70f910
I20260926 12:39:26.975436 21893 ts_tablet_manager.cc:1939] T 4caff133fff147c1b2fbb1856d70f910 P 95d8270f836f4916bae1e3ef6bfb17cc: Deleting consensus metadata
W20260926 12:39:26.975835 21377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:43491 (error 108)
I20260926 12:39:26.976114 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 21587
I20260926 12:39:26.976224 21378 catalog_manager.cc:4841] TS 3a5b19c642cb4719aba25dcec943ad1b (127.10.224.130:35991): tablet 4caff133fff147c1b2fbb1856d70f910 (table test-table [id=dc9818ddafe94256b15e741e77bb4376]) successfully deleted
I20260926 12:39:26.978138 21376 catalog_manager.cc:4841] TS 95d8270f836f4916bae1e3ef6bfb17cc (127.10.224.131:44507): tablet 4caff133fff147c1b2fbb1856d70f910 (table test-table [id=dc9818ddafe94256b15e741e77bb4376]) successfully deleted
I20260926 12:39:26.981498 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:39:26.981096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:43491 (local address 127.0.0.1:37798)
0926 12:39:26.981286 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 12:39:26.981395 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:43491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 12:39:26.981787 21377 catalog_manager.cc:4584] TS d1dde06831604c0c963d109f6522165c (127.10.224.129:43491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4caff133fff147c1b2fbb1856d70f910: Network error: Client connection negotiation failed: client connection to 127.10.224.129:43491: connect: Connection refused (error 111)
I20260926 12:39:26.983389 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 21721
I20260926 12:39:26.989948 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 21359
2026-09-26T12:39:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2396 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:39:27 dist-test-slave-m3pj krb5kdc[21897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:39:27 dist-test-slave-m3pj krb5kdc[21897](info): set up 2 sockets
Sep 26 12:39:27 dist-test-slave-m3pj krb5kdc[21897](info): commencing operation
krb5kdc: starting...
W20260926 12:39:28.057336 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:39:28 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:39:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.224.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.224.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/rangeradmin_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/rangeradmin_127.10.224.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.224.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.224.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/rangerlookup_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/rangerlookup_127.10.224.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.224.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.224.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/HTTP_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/HTTP_127.10.224.148@KRBTEST.COM.keytab.
I20260926 12:39:28.259299 11138 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrbzlsq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/postgres -l logfile start
2026-09-26 12:39:31.302 UTC [21927] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 12:39:31.302 UTC [21927] LOG: listening on IPv4 address "127.10.224.148", port 60701
2026-09-26 12:39:31.307 UTC [21927] LOG: listening on Unix socket "/tmp/.s.PGSQL.60701"
2026-09-26 12:39:31.346 UTC [21930] LOG: database system was shut down at 2026-09-26 12:39:29 UTC
2026-09-26 12:39:31.352 UTC [21927] LOG: database system is ready to accept connections
I20260926 12:39:32.401165 11138 mini_postgres.cc:96] Postgres bound to 60701
2026-09-26 12:39:32.408 UTC [21938] FATAL: database "slave" does not exist
127.10.224.148:60701 - accepting connections
I20260926 12:39:32.408783 11138 mini_ranger.cc:161] Starting Ranger...
I20260926 12:39:32.424697 11138 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 12:39:33.020133 11138 mini_ranger.cc:87] Created ranger Postgres database
I20260926 12:39:33.020248 11138 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-admin
2026-09-26 12:39:33,330 [I] DB FLAVOR :POSTGRES
2026-09-26 12:39:33,330 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:39:33,330 [I] Checking connection..
2026-09-26 12:39:33,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 12:39:33,595 [I] Checking connection passed.
2026-09-26 12:39:33,596 [I] --------- Verifying version history table ---------
2026-09-26 12:39:33,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:39:33,869 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:39:33,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:39:34,148 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:39:34,149 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 12:39:34,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 12:39:34.418 UTC [22002] WARNING: there is no transaction in progress
2026-09-26 12:39:34,425 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 12:39:34,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:39:34,694 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 12:39:34,695 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:39:34,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 12:39:34,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 12:39:35,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m3pj.c.gcp-upstream.internal','N') ;"
2026-09-26 12:39:35,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:39:35,767 [I] Table x_portal_user does not exist in database ranger
2026-09-26 12:39:35,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:39:36,039 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 12:39:36,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:39:36,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:39:36,550 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 12:39:36,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 12:39:37.688 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.710 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.729 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.748 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.767 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.789 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.860 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.869 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.886 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.899 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.911 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.928 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.940 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.949 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.957 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:37.966 UTC [22110] WARNING: there is no transaction in progress
2026-09-26 12:39:38,832 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 12:39:38,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 12:39:39,093 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 12:39:39,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:39:39,399 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 12:39:39,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:39:39,687 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 12:39:39,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:39:39,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:39:40,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m3pj.c.gcp-upstream.internal';"
2026-09-26 12:39:40,470 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 12:39:40,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 12:39:40,732 [I] DB_PATCHES have already been applied
I20260926 12:39:40.738713 11138 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrbzlsq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 12:39:40.738798 11138 mini_ranger.cc:193] Using host: 127.10.224.148
I20260926 12:39:40.742702 11138 mini_ranger.cc:227] Ranger admin URL: http://127.10.224.148:57707
Sep 26, 2026 12:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 12:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 12:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 12:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 12:39:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 12:39:41.896937 11138 mini_ranger.cc:160] Time spent starting Ranger: real 9.488s user 0.000s sys 0.011s
Sep 26, 2026 12:40:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 12:40:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 12:40:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 12:40:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 12:40:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 12:40:06.234395 11138 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.224.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.224.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/rangerkms_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/rangerkms_127.10.224.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.224.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.224.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.224.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/HTTP_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.224.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/HTTP_127.10.224.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 12:40:06 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426406, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 12:40:06.332572 11138 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 12:40:06.343327 11138 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 12:40:07.546541 11138 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 12:40:07.546643 11138 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms
I20260926 12:40:07.546763 11138 mini_ranger_kms.cc:227] Using postgres at 127.10.224.148:60701
2026-09-26 12:40:07,769 [I] DB FLAVOR :POSTGRES
2026-09-26 12:40:07,769 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:40:07,769 [I] Checking connection
2026-09-26 12:40:07,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 12:40:08,023 [I] connection success
2026-09-26 12:40:08,024 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 12:40:08,024 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 12:40:08,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 12:40:08,281 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 12:40:08,281 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:40:08,282 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 12:40:08,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:60701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 12:40:08,571 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 12:40:09.137030 11138 mini_ranger_kms.cc:310] Created kms service
I20260926 12:40:09.392755 11138 mini_ranger_kms.cc:326] Created kudu user
I20260926 12:40:09.449735 11138 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 12:40:09 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426406, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:40:09.921054 11138 mini_ranger_kms.cc:384] Added ranger policy
I20260926 12:40:09.921231 11138 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 12:40:09.921274 11138 mini_ranger_kms.cc:240] Using host: 127.10.224.148
I20260926 12:40:09.925329 11138 mini_ranger_kms.cc:276] Ranger KMS PID: 22291
I20260926 12:40:09.925496 11138 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.224.148:54713
Sep 26, 2026 12:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 12:40:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 12:40:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 12:40:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 12:40:12.225008 11138 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.892s user 0.030s sys 0.176s
I20260926 12:40:12.225165 11138 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 12:40:12.225289 11138 mini_ranger_kms.cc:401] 127.10.224.148:54713/kms/v1/keys
Sep 26 12:40:12 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426412, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.224.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 12:40:12 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426412, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.224.148@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
Sep 26 12:40:13 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426412, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.224.148@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
Sep 26 12:40:15 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:40:15.496076 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38875
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:38875
--ranger_config_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:54713/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:40:15.636221 22341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:40:15.636564 22341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:40:15.636606 22341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:40:15.641012 22341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:40:15.641095 22341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:40:15.641116 22341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:40:15.641135 22341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:40:15.641151 22341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:40:15.647285 22341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.224.148:54713/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:38875
--ranger_config_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:40:15.648587 22341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:40:15.649773 22341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:40:15.661120 22349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:40:15.661149 22346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:40:15.661135 22347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:40:15.661705 22341 server_base.cc:1034] running on GCE node
I20260926 12:40:15.662357 22341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:40:15.663753 22341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:40:15.664966 22341 hybrid_clock.cc:648] HybridClock initialized: now 1790426415664951 us; error 48 us; skew 500 ppm
Sep 26 12:40:15 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426415, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:40:15.668251 22341 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:40:15.669282 22341 webserver.cc:460] Webserver started at http://127.10.224.190:38531/ using document root <none> and password file <none>
I20260926 12:40:15.669919 22341 fs_manager.cc:359] Metadata directory not provided
I20260926 12:40:15.670010 22341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:40:15.670251 22341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:40:15 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426415, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:40:15.736284 22341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "0702b8d1ba6c4ecebba5315e6cec09c1"
format_stamp: "Formatted at 2026-09-26 12:40:15 on dist-test-slave-m3pj"
server_key: "c2e4a618aec939d1c2519aacf4281c8b"
server_key_iv: "f0a9a0b32d21cda6111e79c7cd6613ea"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:15.737035 22341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "0702b8d1ba6c4ecebba5315e6cec09c1"
format_stamp: "Formatted at 2026-09-26 12:40:15 on dist-test-slave-m3pj"
server_key: "c2e4a618aec939d1c2519aacf4281c8b"
server_key_iv: "f0a9a0b32d21cda6111e79c7cd6613ea"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:15.742215 22341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 12:40:15.760996 22356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:40:15.762804 22341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:40:15.762964 22341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "0702b8d1ba6c4ecebba5315e6cec09c1"
format_stamp: "Formatted at 2026-09-26 12:40:15 on dist-test-slave-m3pj"
server_key: "c2e4a618aec939d1c2519aacf4281c8b"
server_key_iv: "f0a9a0b32d21cda6111e79c7cd6613ea"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:15.763111 22341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:40:15.792897 22341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:40:15.796892 22341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:40:15.797137 22341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:40:15.805289 22341 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:38875
I20260926 12:40:15.805295 22408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:38875 every 8 connection(s)
I20260926 12:40:15.806522 22341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:40:15.809718 22409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:40:15.816133 22409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1: Bootstrap starting.
I20260926 12:40:15.816627 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 22341
I20260926 12:40:15.816766 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:40:15.817073 11138 external_mini_cluster.cc:1428] Setting key e8ce8c3284e313fbe87bb086de0236a1
I20260926 12:40:15.819270 22409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1: Neither blocks nor log segments found. Creating new log.
I20260926 12:40:15.820293 22409 log.cc:824] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1: Log is configured to *not* fsync() on all Append() calls
I20260926 12:40:15.823252 22409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1: No bootstrap required, opened a new log
I20260926 12:40:15.828236 22409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0702b8d1ba6c4ecebba5315e6cec09c1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38875 } }
Sep 26 12:40:15 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:40:15.828537 22409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:40:15.828616 22409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0702b8d1ba6c4ecebba5315e6cec09c1, State: Initialized, Role: FOLLOWER
I20260926 12:40:15.829164 22409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0702b8d1ba6c4ecebba5315e6cec09c1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38875 } }
I20260926 12:40:15.829370 22409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:40:15.829481 22409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:40:15.829623 22409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:40:15.830786 22409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0702b8d1ba6c4ecebba5315e6cec09c1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38875 } }
I20260926 12:40:15.831182 22409 leader_election.cc:302] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0702b8d1ba6c4ecebba5315e6cec09c1; no voters:
I20260926 12:40:15.831614 22409 leader_election.cc:288] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:40:15.832196 22414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:40:15.832741 22414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [term 1 LEADER]: Becoming Leader. State: Replica: 0702b8d1ba6c4ecebba5315e6cec09c1, State: Running, Role: LEADER
I20260926 12:40:15.833139 22414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0702b8d1ba6c4ecebba5315e6cec09c1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38875 } }
I20260926 12:40:15.833194 22409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:40:15.835037 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:15.818650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.10.224.190:38875)
0926 12:40:15.819182 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:15.819204 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:15.819252 (+ 48us) server_negotiation.cc:407] Connection header received
0926 12:40:15.820260 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:15.820287 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:15.820780 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:15.821169 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:40:15.822192 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:15.823826 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:15.825124 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:15.825862 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:15.826027 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:15.829326 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:40:15.829345 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:40:15.829366 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:40:15.829403 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:40:15.831612 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:15.832185 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:15.832191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:15.832198 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:15.832347 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:15.832668 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:15.832671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:15.832673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:15.833220 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:40:15.833393 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:15.833742 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 12:40:15.834037 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":169,"threads_started":1}
I20260926 12:40:15.838052 22416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0702b8d1ba6c4ecebba5315e6cec09c1. Latest consensus state: current_term: 1 leader_uuid: "0702b8d1ba6c4ecebba5315e6cec09c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0702b8d1ba6c4ecebba5315e6cec09c1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38875 } } }
I20260926 12:40:15.838207 22416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [sys.catalog]: This master's current role is: LEADER
I20260926 12:40:15.838584 22429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:40:15.839041 22409 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:40:15.840188 22415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0702b8d1ba6c4ecebba5315e6cec09c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0702b8d1ba6c4ecebba5315e6cec09c1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38875 } } }
I20260926 12:40:15.840303 22415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1 [sys.catalog]: This master's current role is: LEADER
I20260926 12:40:15.845330 22429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:40:15.853696 22429 catalog_manager.cc:1347] Generated new cluster ID: b01548fb3f014ec3b9e5517380be6543
I20260926 12:40:15.853854 22429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:40:15.872092 22429 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:40:15.873607 22429 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:40:15.881443 22429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1: Generated new TSK 0
I20260926 12:40:15.882529 22429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:40:16.617345 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:54713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38875
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:40:16.772447 22454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:40:16.772935 22454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:40:16.773054 22454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:40:16.778364 22454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:40:16.778529 22454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:40:16.778648 22454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:40:16.785892 22454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.224.148:54713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:40:16.787093 22454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:40:16.788308 22454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:40:16.801270 22461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:40:16.801370 22462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:40:16.801287 22464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:40:16.802594 22454 server_base.cc:1034] running on GCE node
I20260926 12:40:16.803113 22454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:40:16.803997 22454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:40:16.805181 22454 hybrid_clock.cc:648] HybridClock initialized: now 1790426416805149 us; error 47 us; skew 500 ppm
Sep 26 12:40:16 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426416, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:40:16.809309 22454 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:40:16.810658 22454 webserver.cc:460] Webserver started at http://127.10.224.129:36109/ using document root <none> and password file <none>
I20260926 12:40:16.811558 22454 fs_manager.cc:359] Metadata directory not provided
I20260926 12:40:16.811725 22454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:40:16.812036 22454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:40:16 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426416, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:40:16.832257 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "6e47c57f07ba4aa19a5c7494ae386d64"
format_stamp: "Formatted at 2026-09-26 12:40:16 on dist-test-slave-m3pj"
server_key: "98172ad037832ef87930cafc7a675312"
server_key_iv: "14a3c08d0dace141f91d6fb11ce664f5"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:16.833105 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "6e47c57f07ba4aa19a5c7494ae386d64"
format_stamp: "Formatted at 2026-09-26 12:40:16 on dist-test-slave-m3pj"
server_key: "98172ad037832ef87930cafc7a675312"
server_key_iv: "14a3c08d0dace141f91d6fb11ce664f5"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:16.841289 22454 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260926 12:40:16.871999 22471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:40:16.874833 22454 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260926 12:40:16.875001 22454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "6e47c57f07ba4aa19a5c7494ae386d64"
format_stamp: "Formatted at 2026-09-26 12:40:16 on dist-test-slave-m3pj"
server_key: "98172ad037832ef87930cafc7a675312"
server_key_iv: "14a3c08d0dace141f91d6fb11ce664f5"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:16.875170 22454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:40:16.899060 22454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:40:16.903359 22454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:40:16.903667 22454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:40:16.904456 22454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:40:16.905613 22454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:40:16.905714 22454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:40:16.905800 22454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:40:16.905853 22454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:40:16.917652 22454 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:44241
I20260926 12:40:16.918955 22454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:40:16.928792 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 22454
I20260926 12:40:16.928933 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:40:16.929258 11138 external_mini_cluster.cc:1428] Setting key b23d00fa1da904d2531ae0d6504d7938
I20260926 12:40:16.929867 22586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:44241 every 8 connection(s)
Sep 26 12:40:16 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426416, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:40:16.949744 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:16.928357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:44709 (local address 127.10.224.190:38875)
0926 12:40:16.933037 (+ 4680us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:16.933042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:16.934627 (+ 1585us) server_negotiation.cc:407] Connection header received
0926 12:40:16.934681 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:16.934684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:16.934741 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:16.934826 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:40:16.936765 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:16.937623 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:16.939424 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:16.939900 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:16.940009 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:16.944736 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:40:16.944742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:40:16.944745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:40:16.944781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:40:16.946964 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:16.947797 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:16.947801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:16.947803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:16.947899 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:16.948278 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:16.948282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:16.948283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:16.948553 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:40:16.948664 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:16.949297 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 12:40:16.949440 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4571,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:40:16.950768 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:16.927072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38875 (local address 127.10.224.129:44709)
0926 12:40:16.930832 (+ 3760us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:16.933136 (+ 2304us) client_negotiation.cc:174] Beginning negotiation
0926 12:40:16.934298 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:40:16.935224 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:40:16.935235 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:40:16.935656 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:40:16.936541 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:16.936566 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:16.937794 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:16.937797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:16.939265 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:16.939279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:16.940268 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:16.940271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:16.940426 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:16.942387 (+ 1961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:40:16.942419 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:40:16.944436 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:40:16.947154 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:40:16.947162 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:40:16.947185 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:40:16.947658 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:40:16.947997 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:40:16.947999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:40:16.948001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:40:16.948149 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:40:16.948732 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:40:16.948739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:40:16.949019 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 12:40:16.949244 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 12:40:16.949613 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3458,"thread_start_us":162,"threads_started":1}
I20260926 12:40:16.952813 22587 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38875
I20260926 12:40:16.953119 22587 heartbeater.cc:464] Registering TS with master...
I20260926 12:40:16.953670 22587 heartbeater.cc:511] Master 127.10.224.190:38875 requested a full tablet report, sending...
I20260926 12:40:16.955506 22373 ts_manager.cc:194] Registered new tserver with Master: 6e47c57f07ba4aa19a5c7494ae386d64 (127.10.224.129:44241)
I20260926 12:40:16.957316 22373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:44709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:40:16.972545 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:54713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38875
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:40:16 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426416, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 12:40:17.112939 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:40:17.113278 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:40:17.113389 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:40:17.118269 22593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:40:17.118382 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:40:17.118491 22593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:40:17.124632 22593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.224.148:54713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:40:17.125849 22593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:40:17.126973 22593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:40:17.138795 22599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:40:17.139050 22593 server_base.cc:1034] running on GCE node
W20260926 12:40:17.138795 22602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:40:17.138806 22600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:40:17.139849 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:40:17.140799 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:40:17.142081 22593 hybrid_clock.cc:648] HybridClock initialized: now 1790426417142041 us; error 64 us; skew 500 ppm
Sep 26 12:40:17 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426417, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:40:17.145511 22593 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:40:17.146665 22593 webserver.cc:460] Webserver started at http://127.10.224.130:36865/ using document root <none> and password file <none>
I20260926 12:40:17.147286 22593 fs_manager.cc:359] Metadata directory not provided
I20260926 12:40:17.147354 22593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:40:17.147652 22593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:40:17 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426417, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:40:17.163488 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "04d3b4466ad840e0a6b0a88753cabe08"
format_stamp: "Formatted at 2026-09-26 12:40:17 on dist-test-slave-m3pj"
server_key: "d2b0bca6d9512b0edfcc28e40d76ba78"
server_key_iv: "36fd5a0ab6d8fa1f557494782368e46b"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:17.164570 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "04d3b4466ad840e0a6b0a88753cabe08"
format_stamp: "Formatted at 2026-09-26 12:40:17 on dist-test-slave-m3pj"
server_key: "d2b0bca6d9512b0edfcc28e40d76ba78"
server_key_iv: "36fd5a0ab6d8fa1f557494782368e46b"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:17.177500 22593 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.004s
I20260926 12:40:17.200001 22609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:40:17.202023 22593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:40:17.202195 22593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "04d3b4466ad840e0a6b0a88753cabe08"
format_stamp: "Formatted at 2026-09-26 12:40:17 on dist-test-slave-m3pj"
server_key: "d2b0bca6d9512b0edfcc28e40d76ba78"
server_key_iv: "36fd5a0ab6d8fa1f557494782368e46b"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:17.202368 22593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:40:17.223148 22593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:40:17.227500 22593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:40:17.227828 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:40:17.228672 22593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:40:17.229888 22593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:40:17.229990 22593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 12:40:17.230089 22593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:40:17.230149 22593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:40:17.242426 22593 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:35493
I20260926 12:40:17.242579 22722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:35493 every 8 connection(s)
I20260926 12:40:17.243702 22593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:40:17.248518 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 22593
I20260926 12:40:17.248688 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:40:17.249034 11138 external_mini_cluster.cc:1428] Setting key f89a968cf37b0124f5e602ce275c9052
Sep 26 12:40:17 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426417, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:40:17.268311 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.245873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:39507 (local address 127.10.224.190:38875)
0926 12:40:17.246014 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:17.246019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:17.248082 (+ 2063us) server_negotiation.cc:407] Connection header received
0926 12:40:17.249369 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:17.249375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:17.249438 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:17.249512 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:40:17.251844 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.252619 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.254471 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.254859 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.254962 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.263258 (+ 8296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:40:17.263263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:40:17.263267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:40:17.263307 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:40:17.265350 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:17.266237 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:17.266241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:17.266243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:17.266356 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:17.266897 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:17.266900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:17.266902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:17.267116 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:40:17.267242 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:17.267947 (+ 705us) server_negotiation.cc:299] Negotiation successful
0926 12:40:17.268101 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:40:17.269475 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.246432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38875 (local address 127.10.224.130:39507)
0926 12:40:17.247842 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:17.247905 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 12:40:17.249059 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:40:17.250011 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:40:17.250022 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:40:17.250428 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:40:17.251217 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.251242 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.252794 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.252798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.254305 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.254318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.255200 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.255204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.255415 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.256238 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:40:17.256263 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:40:17.262930 (+ 6667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:40:17.265565 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:40:17.265573 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:40:17.265598 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:40:17.266049 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:40:17.266500 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:40:17.266504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:40:17.266506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:40:17.266770 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:40:17.267242 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:40:17.267253 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:40:17.267630 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 12:40:17.267889 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 12:40:17.268261 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"thread_start_us":160,"threads_started":1}
I20260926 12:40:17.270912 22723 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38875
I20260926 12:40:17.271241 22723 heartbeater.cc:464] Registering TS with master...
I20260926 12:40:17.271898 22723 heartbeater.cc:511] Master 127.10.224.190:38875 requested a full tablet report, sending...
I20260926 12:40:17.273259 22373 ts_manager.cc:194] Registered new tserver with Master: 04d3b4466ad840e0a6b0a88753cabe08 (127.10.224.130:35493)
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:40:17.274091 22373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:39507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:40:17.296064 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:54713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38875
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:40:17.469112 22728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:40:17.469619 22728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:40:17.469734 22728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 26 12:40:17 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426416, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
W20260926 12:40:17.476927 22728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:40:17.477092 22728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:40:17.477234 22728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:40:17.484591 22728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.224.148:54713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:40:17.486451 22728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:40:17.487985 22728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:40:17.501340 22734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:40:17.502216 22733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:40:17.502679 22736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:40:17.504236 22728 server_base.cc:1034] running on GCE node
I20260926 12:40:17.504995 22728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:40:17.505972 22728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:40:17.509285 22728 hybrid_clock.cc:648] HybridClock initialized: now 1790426417509184 us; error 96 us; skew 500 ppm
Sep 26 12:40:17 dist-test-slave-m3pj krb5kdc[21897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426417, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:40:17.513676 22728 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:40:17.514966 22728 webserver.cc:460] Webserver started at http://127.10.224.131:35517/ using document root <none> and password file <none>
I20260926 12:40:17.515884 22728 fs_manager.cc:359] Metadata directory not provided
I20260926 12:40:17.516034 22728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:40:17.516320 22728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:40:17 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426417, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:40:17.543833 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "d4754310f2bc4cb091ee690ffe3b3dcc"
format_stamp: "Formatted at 2026-09-26 12:40:17 on dist-test-slave-m3pj"
server_key: "9781ef48e3a4af4450509092d5345ad6"
server_key_iv: "a678b9f9de92ef588c87a17a338bbff9"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:17.546233 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "d4754310f2bc4cb091ee690ffe3b3dcc"
format_stamp: "Formatted at 2026-09-26 12:40:17 on dist-test-slave-m3pj"
server_key: "9781ef48e3a4af4450509092d5345ad6"
server_key_iv: "a678b9f9de92ef588c87a17a338bbff9"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:17.565176 22728 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.004s sys 0.000s
I20260926 12:40:17.603878 22743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:40:17.605968 22728 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260926 12:40:17.606244 22728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "d4754310f2bc4cb091ee690ffe3b3dcc"
format_stamp: "Formatted at 2026-09-26 12:40:17 on dist-test-slave-m3pj"
server_key: "9781ef48e3a4af4450509092d5345ad6"
server_key_iv: "a678b9f9de92ef588c87a17a338bbff9"
server_key_version: "kuduclusterkey@0"
I20260926 12:40:17.606514 22728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:40:17.636049 22728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:40:17.640255 22728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:40:17.640494 22728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:40:17.641534 22728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:40:17.642686 22728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:40:17.642755 22728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:40:17.642813 22728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:40:17.642881 22728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 12:40:17.655484 22728 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:41299
I20260926 12:40:17.656877 22728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:40:17.660182 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 22728
I20260926 12:40:17.660312 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:40:17.660651 11138 external_mini_cluster.cc:1428] Setting key bdabc562c98e856e7a7abab8ff1e70fc
I20260926 12:40:17.660617 22856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:41299 every 8 connection(s)
Sep 26 12:40:17 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426417, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:40:17.692139 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.665800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:51877 (local address 127.10.224.190:38875)
0926 12:40:17.665925 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:17.665930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:17.675663 (+ 9733us) server_negotiation.cc:407] Connection header received
0926 12:40:17.677056 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:17.677060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:17.677117 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:17.677171 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:40:17.678952 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.679774 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.681887 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.682242 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.682324 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.687441 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:40:17.687445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:40:17.687448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:40:17.687478 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:40:17.689448 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:17.690160 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:17.690164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:17.690166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:17.690255 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:17.690705 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:17.690708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:17.690710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:17.690955 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:40:17.691030 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:17.691835 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 12:40:17.691972 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:40:17.694188 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.664578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38875 (local address 127.10.224.131:51877)
0926 12:40:17.675401 (+ 10823us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:17.675497 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 12:40:17.676786 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:40:17.677648 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:40:17.677659 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:40:17.678051 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:40:17.678778 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.678798 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.680006 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.680009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.681665 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.681677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.682580 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.682584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.682799 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.683628 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:40:17.683652 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:40:17.687225 (+ 3573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:40:17.689614 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:40:17.689622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:40:17.689644 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:40:17.690036 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:40:17.690366 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:40:17.690368 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:40:17.690370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:40:17.690591 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:40:17.691118 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:40:17.691125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:40:17.691483 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 12:40:17.692696 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0926 12:40:17.693074 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10472,"thread_start_us":168,"threads_started":1}
I20260926 12:40:17.695891 22857 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38875
I20260926 12:40:17.696154 22857 heartbeater.cc:464] Registering TS with master...
I20260926 12:40:17.696765 22857 heartbeater.cc:511] Master 127.10.224.190:38875 requested a full tablet report, sending...
I20260926 12:40:17.699599 22373 ts_manager.cc:194] Registered new tserver with Master: d4754310f2bc4cb091ee690ffe3b3dcc (127.10.224.131:41299)
I20260926 12:40:17.700338 22373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:51877
I20260926 12:40:17.711251 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:40:17.729070 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.713693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.10.224.190:38875)
0926 12:40:17.713825 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:17.713830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:17.718832 (+ 5002us) server_negotiation.cc:407] Connection header received
0926 12:40:17.719074 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:17.719078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:17.719137 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:17.719194 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:40:17.720348 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.721248 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.722552 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.722928 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.723013 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.725079 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:40:17.725084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:40:17.725088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:40:17.725123 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:40:17.726994 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:17.727589 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:17.727594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:17.727596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:17.727696 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:40:17.728032 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:40:17.728035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:40:17.728038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:40:17.728275 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:40:17.728391 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:17.728763 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 12:40:17.728896 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:40:17.734385 22373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:40:17.739817 22373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:40:17.772873 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.755637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:41299 (local address 127.0.0.1:40272)
0926 12:40:17.761480 (+ 5843us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:17.761552 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 12:40:17.761707 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:40:17.764371 (+ 2664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:40:17.764378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:40:17.764404 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:40:17.764560 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.764567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.766148 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.766152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.769459 (+ 3307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.769471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.771236 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.771240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.771381 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.771425 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 12:40:17.772484 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0926 12:40:17.772701 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5602,"thread_start_us":97,"threads_started":1}
I20260926 12:40:17.773931 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.754924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.10.224.131:41299)
0926 12:40:17.762304 (+ 7380us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:17.762320 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:17.762349 (+ 29us) server_negotiation.cc:407] Connection header received
0926 12:40:17.762438 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:17.762444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:17.763975 (+ 1531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:17.764159 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:40:17.764721 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.765998 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.769751 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.770853 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.770983 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.771173 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:17.773600 (+ 2427us) server_negotiation.cc:299] Negotiation successful
0926 12:40:17.773752 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6991,"thread_start_us":139,"threads_started":1}
I20260926 12:40:17.774115 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.757722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:35493 (local address 127.0.0.1:57602)
0926 12:40:17.763741 (+ 6019us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:17.763757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:40:17.763846 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:40:17.769197 (+ 5351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:40:17.769201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:40:17.769221 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:40:17.769333 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.769339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.770941 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.770944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.772218 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.772230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.773581 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.773586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.773715 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.773778 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 12:40:17.773872 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:40:17.773950 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5947,"thread_start_us":115,"threads_started":1}
I20260926 12:40:17.774451 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.758918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.10.224.129:44241)
0926 12:40:17.761154 (+ 2236us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:17.761163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:17.768954 (+ 7791us) server_negotiation.cc:407] Connection header received
0926 12:40:17.769049 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:17.769055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:17.769270 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:17.769416 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:40:17.769878 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.771019 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.772511 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.773389 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.773516 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.773646 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:17.774190 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 12:40:17.774311 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2081,"thread_start_us":142,"threads_started":1}
I20260926 12:40:17.774871 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.758498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44241 (local address 127.0.0.1:60768)
0926 12:40:17.768715 (+ 10217us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:17.768731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:40:17.768838 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:40:17.769578 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:40:17.769582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:40:17.769606 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:40:17.769732 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.769742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.771164 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.771168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.772361 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.772371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.773740 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.773744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.773857 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.773882 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:40:17.773964 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 12:40:17.774057 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10143,"thread_start_us":102,"threads_started":1}
I20260926 12:40:17.777992 22784 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1f7c850b4f425dbc56a165377c4ca2 (DEFAULT_TABLE table=test-table [id=7c6b22d4ed34400b9ef5ccacf5c7df1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:40:17.781630 22784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1f7c850b4f425dbc56a165377c4ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:40:17.782938 22507 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1f7c850b4f425dbc56a165377c4ca2 (DEFAULT_TABLE table=test-table [id=7c6b22d4ed34400b9ef5ccacf5c7df1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:40:17.783979 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.757899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.10.224.130:35493)
0926 12:40:17.768381 (+ 10482us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:17.768402 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:17.768434 (+ 32us) server_negotiation.cc:407] Connection header received
0926 12:40:17.768526 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:17.768533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:17.768814 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:17.769015 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:40:17.769465 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.770794 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.772376 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.773219 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.773347 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.773492 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:17.783656 (+ 10164us) server_negotiation.cc:299] Negotiation successful
0926 12:40:17.783821 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10345,"thread_start_us":114,"threads_started":1}
I20260926 12:40:17.784435 22507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1f7c850b4f425dbc56a165377c4ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:40:17.787509 22638 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1f7c850b4f425dbc56a165377c4ca2 (DEFAULT_TABLE table=test-table [id=7c6b22d4ed34400b9ef5ccacf5c7df1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:40:17.788779 22638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1f7c850b4f425dbc56a165377c4ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:40:17.793463 22876 tablet_bootstrap.cc:492] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Bootstrap starting.
I20260926 12:40:17.797801 22877 tablet_bootstrap.cc:492] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: Bootstrap starting.
I20260926 12:40:17.797825 22876 tablet_bootstrap.cc:654] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Neither blocks nor log segments found. Creating new log.
I20260926 12:40:17.798812 22876 log.cc:824] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Log is configured to *not* fsync() on all Append() calls
I20260926 12:40:17.800500 22877 tablet_bootstrap.cc:654] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: Neither blocks nor log segments found. Creating new log.
I20260926 12:40:17.801330 22876 tablet_bootstrap.cc:492] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: No bootstrap required, opened a new log
I20260926 12:40:17.801550 22877 log.cc:824] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: Log is configured to *not* fsync() on all Append() calls
I20260926 12:40:17.801590 22876 ts_tablet_manager.cc:1397] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 12:40:17.807694 22878 tablet_bootstrap.cc:492] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64: Bootstrap starting.
I20260926 12:40:17.810928 22878 tablet_bootstrap.cc:654] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64: Neither blocks nor log segments found. Creating new log.
I20260926 12:40:17.811141 22877 tablet_bootstrap.cc:492] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: No bootstrap required, opened a new log
I20260926 12:40:17.811465 22877 ts_tablet_manager.cc:1397] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260926 12:40:17.812052 22878 log.cc:824] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64: Log is configured to *not* fsync() on all Append() calls
I20260926 12:40:17.814646 22876 raft_consensus.cc:348] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.816383 22877 raft_consensus.cc:348] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.817200 22876 raft_consensus.cc:374] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:40:17.817319 22876 raft_consensus.cc:729] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4754310f2bc4cb091ee690ffe3b3dcc, State: Initialized, Role: FOLLOWER
I20260926 12:40:17.818061 22876 consensus_queue.cc:260] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.819829 22877 raft_consensus.cc:374] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:40:17.819962 22877 raft_consensus.cc:729] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d3b4466ad840e0a6b0a88753cabe08, State: Initialized, Role: FOLLOWER
I20260926 12:40:17.820608 22877 consensus_queue.cc:260] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.821857 22877 ts_tablet_manager.cc:1428] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20260926 12:40:17.822722 22876 ts_tablet_manager.cc:1428] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260926 12:40:17.822886 22878 tablet_bootstrap.cc:492] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64: No bootstrap required, opened a new log
I20260926 12:40:17.823177 22878 ts_tablet_manager.cc:1397] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260926 12:40:17.823228 22857 heartbeater.cc:503] Master 127.10.224.190:38875 was elected leader, sending a full tablet report...
I20260926 12:40:17.827956 22723 heartbeater.cc:503] Master 127.10.224.190:38875 was elected leader, sending a full tablet report...
I20260926 12:40:17.832226 22878 raft_consensus.cc:348] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.834240 22878 raft_consensus.cc:374] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:40:17.834381 22878 raft_consensus.cc:729] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e47c57f07ba4aa19a5c7494ae386d64, State: Initialized, Role: FOLLOWER
I20260926 12:40:17.835255 22878 consensus_queue.cc:260] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.836752 22878 ts_tablet_manager.cc:1428] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260926 12:40:17.837213 22587 heartbeater.cc:503] Master 127.10.224.190:38875 was elected leader, sending a full tablet report...
Sep 26 12:40:17 dist-test-slave-m3pj krb5kdc[21897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426416, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
W20260926 12:40:17.914059 22858 tablet.cc:2369] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:40:17.925485 22588 tablet.cc:2369] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:40:17.936445 22882 raft_consensus.cc:482] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:40:17.936718 22882 raft_consensus.cc:504] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.938180 22882 leader_election.cc:288] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04d3b4466ad840e0a6b0a88753cabe08 (127.10.224.130:35493), 6e47c57f07ba4aa19a5c7494ae386d64 (127.10.224.129:44241)
I20260926 12:40:17.944366 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.938662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:35493 (local address 127.10.224.131:35219)
0926 12:40:17.939417 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:17.939432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:40:17.939560 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:40:17.940230 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:40:17.940233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:40:17.940254 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:40:17.940382 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.940388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.941464 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.941467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.942608 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.942623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.943905 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.943910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.944023 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.944050 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:40:17.944127 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:40:17.944211 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671}
I20260926 12:40:17.945429 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.939828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:35219 (local address 127.10.224.130:35493)
0926 12:40:17.939974 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:17.939979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:17.939997 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:40:17.940057 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:17.940060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:17.940124 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:17.940212 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:40:17.940541 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.941298 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.942820 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.943599 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.943749 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.943824 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:17.945211 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0926 12:40:17.945304 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:40:17.946192 22671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c850b4f425dbc56a165377c4ca2" candidate_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d3b4466ad840e0a6b0a88753cabe08" is_pre_election: true
I20260926 12:40:17.946564 22671 raft_consensus.cc:2454] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4754310f2bc4cb091ee690ffe3b3dcc in term 0.
I20260926 12:40:17.948387 22746 leader_election.cc:302] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04d3b4466ad840e0a6b0a88753cabe08, d4754310f2bc4cb091ee690ffe3b3dcc; no voters:
I20260926 12:40:17.948766 22882 raft_consensus.cc:2781] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:40:17.948853 22882 raft_consensus.cc:482] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:40:17.948896 22882 raft_consensus.cc:3037] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:40:17.950337 22882 raft_consensus.cc:504] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.951231 22671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c850b4f425dbc56a165377c4ca2" candidate_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d3b4466ad840e0a6b0a88753cabe08"
I20260926 12:40:17.951396 22671 raft_consensus.cc:3037] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:40:17.952814 22671 raft_consensus.cc:2454] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4754310f2bc4cb091ee690ffe3b3dcc in term 1.
I20260926 12:40:17.953238 22746 leader_election.cc:302] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04d3b4466ad840e0a6b0a88753cabe08, d4754310f2bc4cb091ee690ffe3b3dcc; no voters:
I20260926 12:40:17.953549 22882 leader_election.cc:288] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [CANDIDATE]: Term 1 election: Requested vote from peers 04d3b4466ad840e0a6b0a88753cabe08 (127.10.224.130:35493), 6e47c57f07ba4aa19a5c7494ae386d64 (127.10.224.129:44241)
I20260926 12:40:17.953640 22882 raft_consensus.cc:2781] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:40:17.953706 22882 raft_consensus.cc:686] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 1 LEADER]: Becoming Leader. State: Replica: d4754310f2bc4cb091ee690ffe3b3dcc, State: Running, Role: LEADER
I20260926 12:40:17.954084 22882 consensus_queue.cc:237] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } }
I20260926 12:40:17.956280 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.938990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44241 (local address 127.10.224.131:58731)
0926 12:40:17.944585 (+ 5595us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:17.944601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:40:17.944692 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:40:17.945852 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:40:17.945855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:40:17.945877 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:40:17.945959 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.945964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.946983 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.946986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.948077 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:40:17.948088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.955817 (+ 7729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.955821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:40:17.955927 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.955948 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:40:17.956016 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 12:40:17.956100 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5532,"thread_start_us":128,"threads_started":1}
I20260926 12:40:17.956795 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:17.939961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:58731 (local address 127.10.224.129:44241)
0926 12:40:17.940077 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:17.940082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:17.944779 (+ 4697us) server_negotiation.cc:407] Connection header received
0926 12:40:17.944838 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:17.944841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:17.944891 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:17.944952 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:40:17.946093 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.946849 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.954778 (+ 7929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:17.955499 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:17.955644 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:17.955706 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:17.956554 (+ 848us) server_negotiation.cc:299] Negotiation successful
0926 12:40:17.956668 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:40:17.957664 22541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c850b4f425dbc56a165377c4ca2" candidate_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" is_pre_election: true
I20260926 12:40:17.958011 22541 raft_consensus.cc:2454] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4754310f2bc4cb091ee690ffe3b3dcc in term 0.
I20260926 12:40:17.958365 22540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1f7c850b4f425dbc56a165377c4ca2" candidate_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e47c57f07ba4aa19a5c7494ae386d64"
I20260926 12:40:17.958479 22540 raft_consensus.cc:3037] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:40:17.959942 22540 raft_consensus.cc:2454] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4754310f2bc4cb091ee690ffe3b3dcc in term 1.
I20260926 12:40:17.969818 22373 catalog_manager.cc:5495] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc reported cstate change: term changed from 0 to 1, leader changed from <none> to d4754310f2bc4cb091ee690ffe3b3dcc (127.10.224.131). New cstate: current_term: 1 leader_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4754310f2bc4cb091ee690ffe3b3dcc" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 } health_report { overall_health: UNKNOWN } } }
W20260926 12:40:18.005059 22724 tablet.cc:2369] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:40:18.012709 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:18.006527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.10.224.131:41299)
0926 12:40:18.007002 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 12:40:18.007007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:40:18.007023 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:40:18.007119 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:40:18.007122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:40:18.007228 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:40:18.007335 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:40:18.007837 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:18.008658 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:18.010005 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:40:18.010379 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:40:18.010482 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:40:18.010966 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:40:18.011709 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:40:18.011827 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:40:18.012365 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 12:40:18.012439 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260926 12:40:18.022776 22540 raft_consensus.cc:1261] T 8b1f7c850b4f425dbc56a165377c4ca2 P 6e47c57f07ba4aa19a5c7494ae386d64 [term 1 FOLLOWER]: Refusing update from remote peer d4754310f2bc4cb091ee690ffe3b3dcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:40:18.023941 22888 consensus_queue.cc:1035] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e47c57f07ba4aa19a5c7494ae386d64" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:40:18.032717 22671 raft_consensus.cc:1261] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 1 FOLLOWER]: Refusing update from remote peer d4754310f2bc4cb091ee690ffe3b3dcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:40:18.033725 22888 consensus_queue.cc:1035] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "04d3b4466ad840e0a6b0a88753cabe08" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:40:18.051929 22892 mvcc.cc:205] Tried to move back new op lower bound from 7333586608209268736 to 7333586607969046528. Current Snapshot: MvccSnapshot[applied={T|T < 7333586608209268736}]
I20260926 12:40:18.052711 22891 mvcc.cc:205] Tried to move back new op lower bound from 7333586608209268736 to 7333586607969046528. Current Snapshot: MvccSnapshot[applied={T|T < 7333586608209268736}]
I20260926 12:40:18.056299 22894 mvcc.cc:205] Tried to move back new op lower bound from 7333586608209268736 to 7333586607969046528. Current Snapshot: MvccSnapshot[applied={T|T < 7333586608209268736}]
I20260926 12:40:18.065752 22373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:40:18.065971 22373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:40:18.069092 22373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0702b8d1ba6c4ecebba5315e6cec09c1: Sending DeleteTablet for 3 replicas of tablet 8b1f7c850b4f425dbc56a165377c4ca2
I20260926 12:40:18.070184 22638 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c850b4f425dbc56a165377c4ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:40:18 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:57602
I20260926 12:40:18.070989 22507 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c850b4f425dbc56a165377c4ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:40:18 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:60768
I20260926 12:40:18.071949 22784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1f7c850b4f425dbc56a165377c4ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:40:18 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:40272
I20260926 12:40:18.073251 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 22454
I20260926 12:40:18.078519 22902 tablet_replica.cc:330] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: stopping tablet replica
I20260926 12:40:18.084467 22900 tablet_replica.cc:330] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: stopping tablet replica
I20260926 12:40:18.085093 22902 raft_consensus.cc:2229] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:40:18.085563 22900 raft_consensus.cc:2229] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:40:18.086066 22900 raft_consensus.cc:2258] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:40:18.086642 22902 raft_consensus.cc:2258] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:40:18.088894 22900 ts_tablet_manager.cc:1905] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:40:18.090513 22902 ts_tablet_manager.cc:1905] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:40:18.096176 22900 ts_tablet_manager.cc:1918] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:40:18.097342 22902 ts_tablet_manager.cc:1918] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 12:40:18.097877 22358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:44241 (error 108)
I20260926 12:40:18.098978 22900 log.cc:1196] T 8b1f7c850b4f425dbc56a165377c4ca2 P 04d3b4466ad840e0a6b0a88753cabe08: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/8b1f7c850b4f425dbc56a165377c4ca2
I20260926 12:40:18.099282 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 22593
I20260926 12:40:18.100150 22902 log.cc:1196] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/8b1f7c850b4f425dbc56a165377c4ca2
I20260926 12:40:18.100646 22902 ts_tablet_manager.cc:1939] T 8b1f7c850b4f425dbc56a165377c4ca2 P d4754310f2bc4cb091ee690ffe3b3dcc: Deleting consensus metadata
I20260926 12:40:18.102191 22360 catalog_manager.cc:4841] TS d4754310f2bc4cb091ee690ffe3b3dcc (127.10.224.131:41299): tablet 8b1f7c850b4f425dbc56a165377c4ca2 (table test-table [id=7c6b22d4ed34400b9ef5ccacf5c7df1d]) successfully deleted
I20260926 12:40:18.103442 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:18.103077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44241 (local address 127.0.0.1:60778)
0926 12:40:18.103271 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:18.103347 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:44241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260926 12:40:18.103830 22358 catalog_manager.cc:4584] TS 6e47c57f07ba4aa19a5c7494ae386d64 (127.10.224.129:44241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b1f7c850b4f425dbc56a165377c4ca2: Network error: Client connection negotiation failed: client connection to 127.10.224.129:44241: connect: Connection refused (error 111)
I20260926 12:40:18.107159 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 22728
I20260926 12:40:18.108101 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:40:18.107404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:35493 (local address 127.0.0.1:57612)
0926 12:40:18.107921 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 12:40:18.107990 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.130:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400}
I20260926 12:40:18.116158 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 22341
I20260926 12:40:18.124851 11138 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 12:40:18.460824 11138 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 12:40:18.460987 11138 mini_ranger.cc:66] Stopping Ranger...
I20260926 12:40:18.806864 11138 mini_ranger.cc:68] Stopped Ranger
2026-09-26 12:40:18.807 UTC [21927] LOG: received smart shutdown request
2026-09-26 12:40:18.811 UTC [21927] LOG: background worker "logical replication launcher" (PID 21936) exited with exit code 1
2026-09-26 12:40:18.811 UTC [21931] LOG: shutting down
2026-09-26 12:40:18.991 UTC [21927] LOG: database system is shut down
2026-09-26T12:40:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52127 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:40:19 dist-test-slave-m3pj krb5kdc[22912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:40:19 dist-test-slave-m3pj krb5kdc[22912](info): set up 2 sockets
Sep 26 12:40:19 dist-test-slave-m3pj krb5kdc[22912](info): commencing operation
krb5kdc: starting...
W20260926 12:40:20.180733 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:40:20 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:40:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.224.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.224.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/rangeradmin_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/rangeradmin_127.10.224.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.224.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.224.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/rangerlookup_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/rangerlookup_127.10.224.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.224.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.224.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/HTTP_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/HTTP_127.10.224.148@KRBTEST.COM.keytab.
I20260926 12:40:20.370939 11138 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrbzlsq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/postgres -l logfile start
2026-09-26 12:40:23.153 UTC [22942] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 12:40:23.153 UTC [22942] LOG: listening on IPv4 address "127.10.224.148", port 57961
2026-09-26 12:40:23.158 UTC [22942] LOG: listening on Unix socket "/tmp/.s.PGSQL.57961"
2026-09-26 12:40:23.180 UTC [22945] LOG: database system was shut down at 2026-09-26 12:40:21 UTC
2026-09-26 12:40:23.184 UTC [22942] LOG: database system is ready to accept connections
I20260926 12:40:24.179826 11138 mini_postgres.cc:96] Postgres bound to 57961
2026-09-26 12:40:24.186 UTC [22953] FATAL: database "slave" does not exist
127.10.224.148:57961 - accepting connections
I20260926 12:40:24.186913 11138 mini_ranger.cc:161] Starting Ranger...
I20260926 12:40:24.202405 11138 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 12:40:24.760985 11138 mini_ranger.cc:87] Created ranger Postgres database
I20260926 12:40:24.761054 11138 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-admin
2026-09-26 12:40:25,048 [I] DB FLAVOR :POSTGRES
2026-09-26 12:40:25,048 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:40:25,048 [I] Checking connection..
2026-09-26 12:40:25,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 12:40:25,318 [I] Checking connection passed.
2026-09-26 12:40:25,318 [I] --------- Verifying version history table ---------
2026-09-26 12:40:25,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:40:25,560 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:40:25,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:40:25,834 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:40:25,834 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 12:40:25,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 12:40:26.118 UTC [23017] WARNING: there is no transaction in progress
2026-09-26 12:40:26,125 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 12:40:26,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:40:26,362 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 12:40:26,362 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:40:26,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 12:40:26,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 12:40:26,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m3pj.c.gcp-upstream.internal','N') ;"
2026-09-26 12:40:27,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:40:27,378 [I] Table x_portal_user does not exist in database ranger
2026-09-26 12:40:27,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:40:27,643 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 12:40:27,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:40:27,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:40:28,150 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 12:40:28,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 12:40:29.200 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.219 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.237 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.253 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.272 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.292 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.355 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.362 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.379 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.390 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.402 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.418 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.429 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.436 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.444 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:29.452 UTC [23125] WARNING: there is no transaction in progress
2026-09-26 12:40:30,165 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 12:40:30,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 12:40:30,390 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 12:40:30,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:40:30,655 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 12:40:30,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:40:30,896 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 12:40:30,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:40:31,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:40:31,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m3pj.c.gcp-upstream.internal';"
2026-09-26 12:40:31,635 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 12:40:31,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 12:40:31,875 [I] DB_PATCHES have already been applied
I20260926 12:40:31.881323 11138 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrbzlsq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 12:40:31.881390 11138 mini_ranger.cc:193] Using host: 127.10.224.148
I20260926 12:40:31.884865 11138 mini_ranger.cc:227] Ranger admin URL: http://127.10.224.148:59699
Sep 26, 2026 12:40:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:40:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:40:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 12:40:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 12:40:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 12:40:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 12:40:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 12:40:33.004053 11138 mini_ranger.cc:160] Time spent starting Ranger: real 8.817s user 0.001s sys 0.009s
Sep 26, 2026 12:40:53 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 12:40:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 12:40:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 12:40:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 12:40:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 12:40:55.462777 11138 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.224.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.224.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/rangerkms_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.224.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/rangerkms_127.10.224.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.224.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.224.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.224.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/HTTP_127.10.224.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.224.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/HTTP_127.10.224.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 12:40:55 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426455, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 12:40:55.555305 11138 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 12:40:55.565267 11138 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 12:40:56.609889 11138 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 12:40:56.609977 11138 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms
I20260926 12:40:56.610008 11138 mini_ranger_kms.cc:227] Using postgres at 127.10.224.148:57961
2026-09-26 12:40:56,784 [I] DB FLAVOR :POSTGRES
2026-09-26 12:40:56,784 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:40:56,784 [I] Checking connection
2026-09-26 12:40:56,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 12:40:57,031 [I] connection success
2026-09-26 12:40:57,032 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 12:40:57,032 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 12:40:57,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 12:40:57,273 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 12:40:57,273 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:40:57,274 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 12:40:57,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.224.148:57961/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 12:40:57,569 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 12:40:58.104758 11138 mini_ranger_kms.cc:310] Created kms service
I20260926 12:40:58.331794 11138 mini_ranger_kms.cc:326] Created kudu user
I20260926 12:40:58.380752 11138 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 12:40:58 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426455, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:40:58.810164 11138 mini_ranger_kms.cc:384] Added ranger policy
I20260926 12:40:58.810321 11138 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrbzlsq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrbzlsq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 12:40:58.810343 11138 mini_ranger_kms.cc:240] Using host: 127.10.224.148
I20260926 12:40:58.815116 11138 mini_ranger_kms.cc:276] Ranger KMS PID: 23306
I20260926 12:40:58.815217 11138 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.224.148:47547
Sep 26, 2026 12:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:40:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 12:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 12:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 12:41:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 12:41:01.057869 11138 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.503s user 0.002s sys 0.156s
I20260926 12:41:01.058015 11138 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 12:41:01.058115 11138 mini_ranger_kms.cc:401] 127.10.224.148:47547/kms/v1/keys
Sep 26 12:41:01 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426461, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.224.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 12:41:01 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426461, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.224.148@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
Sep 26 12:41:02 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426461, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.224.148@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
Sep 26 12:41:04 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:41:04.399406 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35363
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:35363
--ranger_config_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:47547/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:04.516995 23356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:04.517364 23356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:04.517447 23356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:04.521792 23356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:04.521916 23356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:04.521952 23356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:04.522010 23356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:04.522053 23356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:04.528075 23356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.224.148:47547/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:35363
--ranger_config_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:04.529304 23356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:04.530403 23356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:04.539888 23361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:04.539903 23362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:04.539896 23364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:04.540413 23356 server_base.cc:1034] running on GCE node
I20260926 12:41:04.541078 23356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:04.542306 23356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:04.543502 23356 hybrid_clock.cc:648] HybridClock initialized: now 1790426464543477 us; error 44 us; skew 500 ppm
Sep 26 12:41:04 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426464, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:04.552513 23356 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:04.553577 23356 webserver.cc:460] Webserver started at http://127.10.224.190:41457/ using document root <none> and password file <none>
I20260926 12:41:04.554173 23356 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:04.554260 23356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:04.554491 23356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:41:04 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426464, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:04.605559 23356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "993d6aea81174ec5b7e7500742c08e94"
format_stamp: "Formatted at 2026-09-26 12:41:04 on dist-test-slave-m3pj"
server_key: "208f54353f1fff0f32db724239b08881"
server_key_iv: "c67a929f8c99b6373d0eec3c570d0cf0"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:04.606238 23356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "993d6aea81174ec5b7e7500742c08e94"
format_stamp: "Formatted at 2026-09-26 12:41:04 on dist-test-slave-m3pj"
server_key: "208f54353f1fff0f32db724239b08881"
server_key_iv: "c67a929f8c99b6373d0eec3c570d0cf0"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:04.610632 23356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:41:04.626089 23371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:04.627774 23356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:41:04.627959 23356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "993d6aea81174ec5b7e7500742c08e94"
format_stamp: "Formatted at 2026-09-26 12:41:04 on dist-test-slave-m3pj"
server_key: "208f54353f1fff0f32db724239b08881"
server_key_iv: "c67a929f8c99b6373d0eec3c570d0cf0"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:04.628103 23356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:04.647748 23356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:04.650943 23356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:04.651224 23356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:04.659696 23356 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:35363
I20260926 12:41:04.659711 23423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:35363 every 8 connection(s)
I20260926 12:41:04.660792 23356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:04.664402 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:04.667972 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 23356
I20260926 12:41:04.668130 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:41:04.668422 11138 external_mini_cluster.cc:1428] Setting key 0aa57e1f1535d52518f15868139aa2ab
I20260926 12:41:04.673213 23424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Bootstrap starting.
I20260926 12:41:04.676045 23424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:04.676940 23424 log.cc:824] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:41:04 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:04.679332 23424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: No bootstrap required, opened a new log
I20260926 12:41:04.683342 23424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } }
I20260926 12:41:04.683678 23424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:04.683768 23424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993d6aea81174ec5b7e7500742c08e94, State: Initialized, Role: FOLLOWER
I20260926 12:41:04.683866 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:04.669951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.10.224.190:35363)
0926 12:41:04.670454 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:04.670469 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:04.670506 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:41:04.671204 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:04.671223 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:04.671641 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:04.671989 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:04.672811 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:04.673870 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:04.674749 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:04.675237 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:04.675408 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:04.678232 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:04.678237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:04.678252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:04.678281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:04.680734 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:04.681192 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:04.681205 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:04.681210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:04.681315 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:04.681604 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:04.681609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:04.681611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:04.682107 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:04.682311 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:04.682616 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 12:41:04.682961 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":149,"threads_started":1}
I20260926 12:41:04.685276 23424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } }
I20260926 12:41:04.685454 23424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:41:04.685540 23424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:41:04.685647 23424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:04.686810 23424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } }
I20260926 12:41:04.687165 23424 leader_election.cc:302] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993d6aea81174ec5b7e7500742c08e94; no voters:
I20260926 12:41:04.687510 23424 leader_election.cc:288] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:41:04.687644 23429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:04.687886 23429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 1 LEADER]: Becoming Leader. State: Replica: 993d6aea81174ec5b7e7500742c08e94, State: Running, Role: LEADER
I20260926 12:41:04.688259 23429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } }
I20260926 12:41:04.689086 23424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:04.689970 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "993d6aea81174ec5b7e7500742c08e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } } }
I20260926 12:41:04.690099 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:04.690333 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993d6aea81174ec5b7e7500742c08e94. Latest consensus state: current_term: 1 leader_uuid: "993d6aea81174ec5b7e7500742c08e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } } }
I20260926 12:41:04.690404 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:04.691289 23424 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:41:04.692754 23438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:04.698855 23438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:04.707746 23438 catalog_manager.cc:1347] Generated new cluster ID: a8d13c34a8764b238b8ed0962943adbd
I20260926 12:41:04.707832 23438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:41:04.732204 23438 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:41:04.734386 23438 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:41:04.749445 23438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Generated new TSK 0
I20260926 12:41:04.750604 23438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:41:05.527441 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:47547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35363
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:05.653324 23469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:05.653642 23469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:05.653781 23469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:05.658394 23469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:05.658505 23469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:05.658600 23469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:41:05.664857 23469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.224.148:47547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:05.666538 23469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:05.668069 23469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:05.679955 23475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:05.680613 23476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:05.680940 23478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:05.681893 23469 server_base.cc:1034] running on GCE node
I20260926 12:41:05.682436 23469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:05.683115 23469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:05.684314 23469 hybrid_clock.cc:648] HybridClock initialized: now 1790426465684278 us; error 52 us; skew 500 ppm
Sep 26 12:41:05 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426465, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:05.687928 23469 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:41:05.689322 23469 webserver.cc:460] Webserver started at http://127.10.224.129:38943/ using document root <none> and password file <none>
I20260926 12:41:05.690284 23469 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:05.690440 23469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:05.690832 23469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:41:05 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426465, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:05.714462 23469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "9ca0329165424eba86d8ec7261c77a66"
format_stamp: "Formatted at 2026-09-26 12:41:05 on dist-test-slave-m3pj"
server_key: "9b794c47af37e8406658d31f1575fdec"
server_key_iv: "b265a040f9cdfc1597cec6718f3f658c"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:05.715824 23469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "9ca0329165424eba86d8ec7261c77a66"
format_stamp: "Formatted at 2026-09-26 12:41:05 on dist-test-slave-m3pj"
server_key: "9b794c47af37e8406658d31f1575fdec"
server_key_iv: "b265a040f9cdfc1597cec6718f3f658c"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:05.725469 23469 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260926 12:41:05.757637 23486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:05.759764 23469 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.003s sys 0.000s
I20260926 12:41:05.759929 23469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "9ca0329165424eba86d8ec7261c77a66"
format_stamp: "Formatted at 2026-09-26 12:41:05 on dist-test-slave-m3pj"
server_key: "9b794c47af37e8406658d31f1575fdec"
server_key_iv: "b265a040f9cdfc1597cec6718f3f658c"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:05.760097 23469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:05.793823 23469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:05.798019 23469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:05.798254 23469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:05.799026 23469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:05.800346 23469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:05.800443 23469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:05.800530 23469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:05.800586 23469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:05.818277 23601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:45243 every 8 connection(s)
I20260926 12:41:05.818588 23469 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:45243
I20260926 12:41:05.820030 23469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:41:05.826264 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 23469
I20260926 12:41:05.826397 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:41:05.826722 11138 external_mini_cluster.cc:1428] Setting key b153666d851dc26a4c72f9353f5fd7c6
Sep 26 12:41:05 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426465, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:05.846464 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:05.823768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:44279 (local address 127.10.224.190:35363)
0926 12:41:05.825392 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:05.825408 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:05.826489 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 12:41:05.831033 (+ 4544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:05.831038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:05.831122 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:05.831214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:05.833335 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:05.834244 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:05.836207 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:05.836582 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:05.836668 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:05.841475 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:05.841480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:05.841484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:05.841519 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:05.843616 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:05.844372 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:05.844377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:05.844379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:05.844479 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:05.844933 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:05.844937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:05.844939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:05.845265 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:05.845356 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:05.846121 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 12:41:05.846270 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":124,"threads_started":1}
I20260926 12:41:05.848609 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:05.824091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35363 (local address 127.10.224.129:44279)
0926 12:41:05.825721 (+ 1630us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:05.825786 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:05.830558 (+ 4772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:05.831688 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:05.831700 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:05.832199 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:05.833059 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:05.833095 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:05.834428 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:05.834432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:05.836022 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:05.836037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:05.836914 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:05.836917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:05.837167 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:05.839081 (+ 1914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:05.839107 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:05.841203 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:05.843790 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:05.843799 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:05.843820 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:05.844236 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:05.844599 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:05.844602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:05.844604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:05.844813 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:05.845429 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:05.845436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:05.845805 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 12:41:05.847060 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0926 12:41:05.847451 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"thread_start_us":140,"threads_started":1}
I20260926 12:41:05.850464 23602 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35363
I20260926 12:41:05.850725 23602 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:05.851338 23602 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:05.853278 23388 ts_manager.cc:194] Registered new tserver with Master: 9ca0329165424eba86d8ec7261c77a66 (127.10.224.129:45243)
I20260926 12:41:05.854914 23388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:44279
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:41:05.895423 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:47547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35363
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:41:05 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426465, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 12:41:06.074005 23608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:06.074501 23608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:06.074611 23608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:06.081424 23608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:06.081616 23608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:06.081779 23608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:41:06.090154 23608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.224.148:47547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:06.091931 23608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:06.093670 23608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:06.108237 23614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:06.108819 23617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:06.109155 23615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:06.110713 23608 server_base.cc:1034] running on GCE node
I20260926 12:41:06.111258 23608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:06.112152 23608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:06.113516 23608 hybrid_clock.cc:648] HybridClock initialized: now 1790426466113444 us; error 91 us; skew 500 ppm
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:06.117511 23608 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:41:06.118780 23608 webserver.cc:460] Webserver started at http://127.10.224.130:36151/ using document root <none> and password file <none>
I20260926 12:41:06.119573 23608 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:06.119719 23608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:06.120016 23608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:06.139137 23608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "c7039f803b044552a810af63bab762a9"
format_stamp: "Formatted at 2026-09-26 12:41:06 on dist-test-slave-m3pj"
server_key: "bd89ce5073d82f0103ce0eb8935cbb53"
server_key_iv: "3e3f9cd1f6ae1effb3a21f179244ce01"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:06.141683 23608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "c7039f803b044552a810af63bab762a9"
format_stamp: "Formatted at 2026-09-26 12:41:06 on dist-test-slave-m3pj"
server_key: "bd89ce5073d82f0103ce0eb8935cbb53"
server_key_iv: "3e3f9cd1f6ae1effb3a21f179244ce01"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:06.147676 23608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 12:41:06.171329 23624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:06.173221 23608 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 12:41:06.173372 23608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "c7039f803b044552a810af63bab762a9"
format_stamp: "Formatted at 2026-09-26 12:41:06 on dist-test-slave-m3pj"
server_key: "bd89ce5073d82f0103ce0eb8935cbb53"
server_key_iv: "3e3f9cd1f6ae1effb3a21f179244ce01"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:06.173509 23608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:06.201097 23608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:06.205477 23608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:06.205787 23608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:06.206593 23608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:06.207862 23608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:06.207978 23608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:06.208086 23608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:06.208153 23608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:06.230661 23608 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:33689
I20260926 12:41:06.232108 23608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:41:06.232410 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 23608
I20260926 12:41:06.232549 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:41:06.232932 11138 external_mini_cluster.cc:1428] Setting key 97a3e47a59f2052b29e42492b9769179
I20260926 12:41:06.236325 23737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:33689 every 8 connection(s)
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:41:06.256919 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:06.240795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:40667 (local address 127.10.224.190:35363)
0926 12:41:06.240966 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:06.240971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:06.244175 (+ 3204us) server_negotiation.cc:407] Connection header received
0926 12:41:06.245464 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:06.245468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:06.245536 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:06.245645 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:06.246980 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:06.247625 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:06.248899 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:06.249304 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:06.249436 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:06.252594 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:06.252597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:06.252600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:06.252629 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:06.254437 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:06.255113 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:06.255116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:06.255118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:06.255217 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:06.255652 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:06.255656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:06.255658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:06.255875 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:06.255992 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:06.256569 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 12:41:06.256720 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:41:06.257812 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:06.239560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35363 (local address 127.10.224.130:40667)
0926 12:41:06.243950 (+ 4390us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:06.244025 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:06.245165 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:06.245895 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:06.245905 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:06.246240 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:06.246791 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:06.246806 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:06.247805 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:06.247808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:06.248746 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:06.248756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:06.249523 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:06.249526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:06.249894 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:06.250675 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:06.250696 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:06.252356 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:06.254625 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:06.254632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:06.254659 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:06.254989 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:06.255324 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:06.255327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:06.255329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:06.255509 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:06.255993 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:06.256002 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:06.256304 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 12:41:06.256511 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 12:41:06.256901 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4099,"thread_start_us":163,"threads_started":1}
I20260926 12:41:06.261016 23738 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35363
I20260926 12:41:06.261373 23738 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:06.262030 23738 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:06.263550 23388 ts_manager.cc:194] Registered new tserver with Master: c7039f803b044552a810af63bab762a9 (127.10.224.130:33689)
I20260926 12:41:06.264420 23388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:40667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:41:06.279855 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:47547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35363
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:06.433859 23743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:06.434199 23743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:06.434348 23743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:06.439805 23743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:06.439899 23743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:06.440004 23743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426465, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:06.447160 23743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.224.148:47547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:06.448576 23743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:06.449931 23743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:06.463219 23751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:06.463502 23748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:06.463219 23749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:06.463917 23743 server_base.cc:1034] running on GCE node
I20260926 12:41:06.464560 23743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:06.465488 23743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:06.466672 23743 hybrid_clock.cc:648] HybridClock initialized: now 1790426466466645 us; error 39 us; skew 500 ppm
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:06.470755 23743 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:41:06.472209 23743 webserver.cc:460] Webserver started at http://127.10.224.131:39389/ using document root <none> and password file <none>
I20260926 12:41:06.473124 23743 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:06.473292 23743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:06.473609 23743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:06.500240 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "8edd7e09c08d4428bdec11181ddc08ad"
format_stamp: "Formatted at 2026-09-26 12:41:06 on dist-test-slave-m3pj"
server_key: "7ffb03fbdd2d59171154f666f7bcdbcc"
server_key_iv: "49ae8f7ec541e35a0c39fcb6de544f7c"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:06.501160 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "8edd7e09c08d4428bdec11181ddc08ad"
format_stamp: "Formatted at 2026-09-26 12:41:06 on dist-test-slave-m3pj"
server_key: "7ffb03fbdd2d59171154f666f7bcdbcc"
server_key_iv: "49ae8f7ec541e35a0c39fcb6de544f7c"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:06.507654 23743 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 12:41:06.530571 23758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:06.532441 23743 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260926 12:41:06.532582 23743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "8edd7e09c08d4428bdec11181ddc08ad"
format_stamp: "Formatted at 2026-09-26 12:41:06 on dist-test-slave-m3pj"
server_key: "7ffb03fbdd2d59171154f666f7bcdbcc"
server_key_iv: "49ae8f7ec541e35a0c39fcb6de544f7c"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:06.532714 23743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:06.581064 23743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:06.585245 23743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:06.585554 23743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:06.586376 23743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:06.587601 23743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:06.587715 23743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:06.587812 23743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:06.587877 23743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:06.601579 23743 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:37569
I20260926 12:41:06.602905 23743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:41:06.604411 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 23743
I20260926 12:41:06.604532 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:41:06.604831 11138 external_mini_cluster.cc:1428] Setting key 55d129d1f707733d3b7edc4cdd96f1e6
I20260926 12:41:06.604794 23871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:37569 every 8 connection(s)
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:06.629216 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:06.607714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:36337 (local address 127.10.224.190:35363)
0926 12:41:06.607832 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:06.607837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:06.609387 (+ 1550us) server_negotiation.cc:407] Connection header received
0926 12:41:06.610839 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:06.610843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:06.610905 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:06.610960 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:06.612825 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:06.613675 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:06.620905 (+ 7230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:06.621260 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:06.621337 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:06.624635 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:06.624640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:06.624642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:06.624673 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:06.626529 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:06.627242 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:06.627246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:06.627248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:06.627350 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:06.627835 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:06.627839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:06.627841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:06.628095 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:06.628202 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:06.628894 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 12:41:06.629036 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:41:06.631731 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:06.607220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35363 (local address 127.10.224.131:36337)
0926 12:41:06.609225 (+ 2005us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:06.609285 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:06.610580 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:06.611458 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:06.611467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:06.611937 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:06.612642 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:06.612664 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:06.613851 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:06.613854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:06.620736 (+ 6882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:06.620755 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:06.621574 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:06.621578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:06.621800 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:06.622585 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:06.622608 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:06.624399 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:06.626680 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:06.626687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:06.626707 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:06.627068 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:06.627467 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:06.627470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:06.627473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:06.627715 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:06.628223 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:06.628230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:06.628587 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 12:41:06.628799 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 12:41:06.630187 (+ 1388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1765,"thread_start_us":503,"threads_started":1}
I20260926 12:41:06.633136 23872 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35363
I20260926 12:41:06.633436 23872 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:06.634214 23872 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:06.635505 23388 ts_manager.cc:194] Registered new tserver with Master: 8edd7e09c08d4428bdec11181ddc08ad (127.10.224.131:37569)
I20260926 12:41:06.636348 23388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:36337
I20260926 12:41:06.642758 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:41:06.643628 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 23469
I20260926 12:41:06.652797 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 23608
I20260926 12:41:06.661268 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 23743
I20260926 12:41:06.667317 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 23356
I20260926 12:41:06.675503 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35363
--webserver_interface=127.10.224.190
--webserver_port=41457
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:35363
--ranger_config_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:47547/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.224.148:47547/kms with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:06.794662 23875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:06.794962 23875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:06.795003 23875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:06.799348 23875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:06.799432 23875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:06.799453 23875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:06.799469 23875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:06.799484 23875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:06.805343 23875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.224.148:47547/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:35363
--ranger_config_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=41457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:06.806497 23875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:06.807639 23875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:06.816699 23880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:06.816938 23875 server_base.cc:1034] running on GCE node
W20260926 12:41:06.816841 23883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:06.816722 23881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:06.817821 23875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:06.819406 23875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:06.820629 23875 hybrid_clock.cc:648] HybridClock initialized: now 1790426466820591 us; error 59 us; skew 500 ppm
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:06.824414 23875 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:06.825706 23875 webserver.cc:460] Webserver started at http://127.10.224.190:41457/ using document root <none> and password file <none>
I20260926 12:41:06.826413 23875 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:06.826517 23875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:41:06 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426466, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:06.892714 23875 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:41:06.895208 23891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:06.896694 23875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:06.896847 23875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "993d6aea81174ec5b7e7500742c08e94"
format_stamp: "Formatted at 2026-09-26 12:41:04 on dist-test-slave-m3pj"
server_key: "208f54353f1fff0f32db724239b08881"
server_key_iv: "c67a929f8c99b6373d0eec3c570d0cf0"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:06.897483 23875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:06.914171 23875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:06.917624 23875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:06.917865 23875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:06.926152 23875 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:35363
I20260926 12:41:06.926156 23943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:35363 every 8 connection(s)
I20260926 12:41:06.927369 23875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:06.932010 23944 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:41:06.933883 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Bootstrap starting.
I20260926 12:41:06.934855 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 23875
I20260926 12:41:06.935356 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:47547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:45243
--local_ip_for_outbound_sockets=127.10.224.129
--tserver_master_addrs=127.10.224.190:35363
--webserver_port=38943
--webserver_interface=127.10.224.129
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.224.148:47547/kms with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:06.957973 23944 log.cc:824] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:06.967381 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:41:06.967912 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Bootstrap complete.
I20260926 12:41:06.973662 23944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } }
I20260926 12:41:06.974217 23944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993d6aea81174ec5b7e7500742c08e94, State: Initialized, Role: FOLLOWER
I20260926 12:41:06.974843 23944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } }
I20260926 12:41:06.974996 23944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:41:06.975090 23944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:41:06.975199 23944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:41:06.977828 23944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } }
I20260926 12:41:06.978233 23944 leader_election.cc:302] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993d6aea81174ec5b7e7500742c08e94; no voters:
I20260926 12:41:06.978593 23944 leader_election.cc:288] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:41:06.978780 23948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:41:06.979084 23948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [term 2 LEADER]: Becoming Leader. State: Replica: 993d6aea81174ec5b7e7500742c08e94, State: Running, Role: LEADER
I20260926 12:41:06.979480 23948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } }
I20260926 12:41:06.980050 23944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:06.980710 23948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "993d6aea81174ec5b7e7500742c08e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } } }
I20260926 12:41:06.980914 23948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:06.981173 23948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993d6aea81174ec5b7e7500742c08e94. Latest consensus state: current_term: 2 leader_uuid: "993d6aea81174ec5b7e7500742c08e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993d6aea81174ec5b7e7500742c08e94" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35363 } } }
I20260926 12:41:06.981303 23948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:06.981606 23954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:06.985146 23954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:06.985725 23944 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:41:06.986027 23954 catalog_manager.cc:1259] Loaded cluster ID: a8d13c34a8764b238b8ed0962943adbd
I20260926 12:41:06.986132 23954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:41:06.988595 23954 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:41:06.989343 23954 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Loaded TSK: 0
I20260926 12:41:06.990664 23954 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:41:07.104435 23946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:07.104960 23946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:07.105094 23946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:07.111605 23946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:07.111788 23946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:07.111936 23946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:41:07.120240 23946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.224.148:47547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:45243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=38943
--tserver_master_addrs=127.10.224.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:07.121889 23946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:07.123454 23946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:07.138794 23981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:07.139276 23946 server_base.cc:1034] running on GCE node
W20260926 12:41:07.139971 23982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:07.140658 23984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:07.141278 23946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:07.142158 23946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:07.145438 23946 hybrid_clock.cc:648] HybridClock initialized: now 1790426467145345 us; error 101 us; skew 500 ppm
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:07.150363 23946 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:41:07.151732 23946 webserver.cc:460] Webserver started at http://127.10.224.129:38943/ using document root <none> and password file <none>
I20260926 12:41:07.152556 23946 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:07.152750 23946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:07.177444 23946 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260926 12:41:07.180557 23992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.182299 23946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:07.182443 23946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "9ca0329165424eba86d8ec7261c77a66"
format_stamp: "Formatted at 2026-09-26 12:41:05 on dist-test-slave-m3pj"
server_key: "9b794c47af37e8406658d31f1575fdec"
server_key_iv: "b265a040f9cdfc1597cec6718f3f658c"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:07.183230 23946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:07.216374 23946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:07.220279 23946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:07.220513 23946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:07.221216 23946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:07.222262 23946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:07.222332 23946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.222400 23946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:07.222426 23946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.234421 23946 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:45243
I20260926 12:41:07.234437 24105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:45243 every 8 connection(s)
I20260926 12:41:07.235797 23946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:41:07.239739 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 23946
I20260926 12:41:07.240262 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:47547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:33689
--local_ip_for_outbound_sockets=127.10.224.130
--tserver_master_addrs=127.10.224.190:35363
--webserver_port=36151
--webserver_interface=127.10.224.130
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.224.148:47547/kms with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:07.271571 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:07.238485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35363 (local address 127.10.224.129:48095)
0926 12:41:07.239019 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:07.239076 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:07.240593 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:07.244070 (+ 3477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:07.244082 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:07.244462 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:07.245257 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:07.245285 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.247486 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.247492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:07.249145 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:07.249160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.250702 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.250708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:07.257543 (+ 6835us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:07.259742 (+ 2199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:07.259774 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:07.264487 (+ 4713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:07.267609 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:07.267617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:07.267635 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:07.268041 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:07.268505 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:07.268508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:07.268510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:07.268701 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:07.269507 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:07.269514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:07.269860 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 12:41:07.270062 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 12:41:07.270435 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":154,"threads_started":1}
I20260926 12:41:07.273662 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:07.238127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:48095 (local address 127.10.224.190:35363)
0926 12:41:07.242100 (+ 3973us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:07.242127 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:07.242182 (+ 55us) server_negotiation.cc:407] Connection header received
0926 12:41:07.242552 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:07.242571 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:07.243078 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:07.243461 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:07.245509 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.247291 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.249317 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.250201 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.250410 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:07.264848 (+ 14438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:07.264858 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:07.264879 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:07.264926 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:07.267387 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:07.268220 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:07.268228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:07.268234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:07.268393 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:07.268809 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:07.268812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:07.268814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:07.269321 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:07.269450 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:07.272279 (+ 2829us) server_negotiation.cc:299] Negotiation successful
0926 12:41:07.272620 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3727,"thread_start_us":122,"threads_started":1}
I20260926 12:41:07.276546 24106 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35363
I20260926 12:41:07.276818 24106 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.277403 24106 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
W20260926 12:41:07.279630 24106 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:35363 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:41:07.279769 24106 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.279960 24106 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:07.280874 24106 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.281068 24106 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
W20260926 12:41:07.281831 24106 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:41:07.407454 24110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:07.407836 24110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:07.407909 24110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:07.413082 24110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:07.413195 24110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:07.413305 24110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:41:07.420325 24110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.224.148:47547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:33689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=36151
--tserver_master_addrs=127.10.224.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:07.421595 24110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:07.422710 24110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:07.435009 24116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:07.435173 24118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:07.435312 24115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:07.435544 24110 server_base.cc:1034] running on GCE node
I20260926 12:41:07.436055 24110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:07.436949 24110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:07.438140 24110 hybrid_clock.cc:648] HybridClock initialized: now 1790426467438109 us; error 48 us; skew 500 ppm
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:07.442097 24110 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:41:07.443490 24110 webserver.cc:460] Webserver started at http://127.10.224.130:36151/ using document root <none> and password file <none>
I20260926 12:41:07.444325 24110 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:07.444451 24110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:07.468477 24110 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:41:07.472106 24126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.473969 24110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:41:07.474157 24110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "c7039f803b044552a810af63bab762a9"
format_stamp: "Formatted at 2026-09-26 12:41:06 on dist-test-slave-m3pj"
server_key: "bd89ce5073d82f0103ce0eb8935cbb53"
server_key_iv: "3e3f9cd1f6ae1effb3a21f179244ce01"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:07.475018 24110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:07.491161 24110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:07.494988 24110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:07.495229 24110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:07.495977 24110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:07.496937 24110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:07.497007 24110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.497072 24110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:07.497128 24110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.508484 24110 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:33689
I20260926 12:41:07.509124 24239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:33689 every 8 connection(s)
I20260926 12:41:07.509830 24110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:41:07.519297 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 24110
I20260926 12:41:07.519924 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.224.148:47547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:37569
--local_ip_for_outbound_sockets=127.10.224.131
--tserver_master_addrs=127.10.224.190:35363
--webserver_port=39389
--webserver_interface=127.10.224.131
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.224.148:47547/kms with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:07.535151 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:07.515782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:48465 (local address 127.10.224.190:35363)
0926 12:41:07.515936 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:07.515940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:07.515957 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:41:07.516008 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:07.516011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:07.516060 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:07.516145 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:07.518728 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.519574 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.523609 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.524096 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.524253 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:07.529834 (+ 5581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:07.529840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:07.529843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:07.529882 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:07.532077 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:07.532981 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:07.532987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:07.532989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:07.533104 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:07.533599 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:07.533603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:07.533604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:07.533878 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:07.533960 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:07.534790 (+ 830us) server_negotiation.cc:299] Negotiation successful
0926 12:41:07.534953 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:41:07.537358 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:07.512935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35363 (local address 127.10.224.130:48465)
0926 12:41:07.513515 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:07.513571 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:07.514435 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:07.517210 (+ 2775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:07.517223 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:07.517683 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:07.518500 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:07.518522 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.521752 (+ 3230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.521759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:07.523396 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:07.523413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.524496 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.524504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:07.524799 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:07.525597 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:07.525629 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:07.529500 (+ 3871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:07.532279 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:07.532287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:07.532310 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:07.532835 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:07.533238 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:07.533241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:07.533243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:07.533472 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:07.534050 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:07.534058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:07.534458 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 12:41:07.535720 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0926 12:41:07.536153 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":168,"threads_started":1}
I20260926 12:41:07.539350 24240 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35363
I20260926 12:41:07.539654 24240 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.540338 24240 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
W20260926 12:41:07.542320 24240 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:35363 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:41:07.542483 24240 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.542719 24240 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:07.543617 24240 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.543810 24240 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
W20260926 12:41:07.544610 24240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:41:07.691425 24243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:07.691838 24243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:07.691908 24243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:07.697403 24243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:07.697515 24243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:07.697624 24243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:41:07.705157 24243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.224.148:47547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:37569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=39389
--tserver_master_addrs=127.10.224.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:07.706657 24243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:07.707890 24243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:07.718946 24249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:07.718972 24252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:07.719353 24250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:07.719734 24243 server_base.cc:1034] running on GCE node
I20260926 12:41:07.720160 24243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:07.720875 24243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:07.722072 24243 hybrid_clock.cc:648] HybridClock initialized: now 1790426467722039 us; error 56 us; skew 500 ppm
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:07.725006 24243 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:41:07.726122 24243 webserver.cc:460] Webserver started at http://127.10.224.131:39389/ using document root <none> and password file <none>
I20260926 12:41:07.726859 24243 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:07.726974 24243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:07.752544 24243 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.001s
I20260926 12:41:07.755362 24260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.756880 24243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:07.757037 24243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "8edd7e09c08d4428bdec11181ddc08ad"
format_stamp: "Formatted at 2026-09-26 12:41:06 on dist-test-slave-m3pj"
server_key: "7ffb03fbdd2d59171154f666f7bcdbcc"
server_key_iv: "49ae8f7ec541e35a0c39fcb6de544f7c"
server_key_version: "kuduclusterkey@0"
I20260926 12:41:07.757642 24243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:07.771900 24243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:07.775322 24243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:07.775611 24243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:07.776278 24243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:07.777148 24243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:07.777220 24243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.777299 24243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:07.777350 24243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:07.788071 24243 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:37569
I20260926 12:41:07.788146 24373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:37569 every 8 connection(s)
I20260926 12:41:07.789512 24243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:41:07.790647 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 24243
Sep 26 12:41:07 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426467, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:07.808260 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:07.791996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:39415 (local address 127.10.224.190:35363)
0926 12:41:07.793623 (+ 1627us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:07.793628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:07.793647 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:41:07.794705 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:07.794708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:07.794752 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:07.794834 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:07.796476 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.797355 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.799173 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.799549 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.799664 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:07.802979 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:07.802983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:07.802986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:07.803018 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:07.804824 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:07.806171 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:07.806175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:07.806176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:07.806271 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:07.806737 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:07.806740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:07.806742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:07.807005 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:07.807095 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:07.807896 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 12:41:07.808042 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":1542,"spinlock_wait_cycles":124544,"thread_start_us":107,"threads_started":1}
I20260926 12:41:07.808437 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:07.791390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.10.224.190:35363)
0926 12:41:07.792290 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:07.792302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:07.794172 (+ 1870us) server_negotiation.cc:407] Connection header received
0926 12:41:07.794219 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:07.794251 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:07.794317 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:07.794412 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:07.796371 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.797545 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.799279 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.799632 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.799708 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:07.803986 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:07.803990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:07.803992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:07.804023 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:07.806471 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:07.807123 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:07.807124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:07.807126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:07.807184 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:07.807515 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:07.807538 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:07.807543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:07.807712 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:07.807942 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:07.808071 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 12:41:07.808195 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":810}
I20260926 12:41:07.810353 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:07.792571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:35363 (local address 127.10.224.131:39415)
0926 12:41:07.793216 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:07.793279 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:07.794454 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:07.795063 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:07.795074 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:07.795477 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:07.796260 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:07.796284 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.797513 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.797517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:07.799010 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:07.799023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:07.799922 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:07.799926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:07.800079 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:07.800772 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:07.800796 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:07.802764 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:07.804973 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:07.804983 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:07.805006 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:07.806027 (+ 1021us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:07.806392 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:07.806396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:07.806398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:07.806626 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:07.807123 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:07.807131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:07.807503 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 12:41:07.808744 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0926 12:41:07.809168 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":137,"threads_started":1}
I20260926 12:41:07.811605 24374 heartbeater.cc:347] Connected to a master server at 127.10.224.190:35363
I20260926 12:41:07.811857 24374 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.812528 24374 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
W20260926 12:41:07.816872 24374 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:35363 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:41:07.817023 24374 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.817225 24374 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:07.818202 24374 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:07.818408 24374 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
W20260926 12:41:07.819255 24374 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 12:41:08 dist-test-slave-m3pj krb5kdc[22912](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426468, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:08.282058 24106 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:08.282431 24106 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:08.283891 23908 ts_manager.cc:194] Registered new tserver with Master: 9ca0329165424eba86d8ec7261c77a66 (127.10.224.129:45243)
I20260926 12:41:08.285635 23908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:48095
I20260926 12:41:08.544876 24240 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:08.545236 24240 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:08.546279 23908 ts_manager.cc:194] Registered new tserver with Master: c7039f803b044552a810af63bab762a9 (127.10.224.130:33689)
I20260926 12:41:08.547096 23908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:48465
Sep 26 12:41:08 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426468, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:08.819509 24374 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:08.819856 24374 heartbeater.cc:511] Master 127.10.224.190:35363 requested a full tablet report, sending...
I20260926 12:41:08.820672 23908 ts_manager.cc:194] Registered new tserver with Master: 8edd7e09c08d4428bdec11181ddc08ad (127.10.224.131:37569)
I20260926 12:41:08.821437 23908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:39415
I20260926 12:41:08.831388 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:41:08.843184 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:08.833079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.10.224.190:35363)
0926 12:41:08.833341 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:08.833346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:08.833561 (+ 215us) server_negotiation.cc:407] Connection header received
0926 12:41:08.833691 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:08.833694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:08.833752 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:08.833857 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:08.834725 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.835641 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.836871 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.837242 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.837357 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:08.839241 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:08.839245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:08.839248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:08.839280 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:08.841247 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:08.841737 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:08.841741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:08.841743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:08.841841 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:08.842166 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:08.842170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:08.842171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:08.842418 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:08.842565 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:08.842847 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 12:41:08.842984 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260926 12:41:08.847757 23908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:41:08.853193 23908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:41:08.885866 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:08.874005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:37569 (local address 127.0.0.1:57550)
0926 12:41:08.875860 (+ 1855us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:08.875873 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:08.875973 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:08.878774 (+ 2801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:08.878777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:08.878796 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:08.878898 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:08.878904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.881519 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.881523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:08.883069 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:08.883081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.884327 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.884331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:08.884457 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:08.884504 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 12:41:08.885501 (+ 997us) client_negotiation.cc:240] Negotiation successful
0926 12:41:08.885694 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1794,"mutex_wait_us":153,"thread_start_us":95,"threads_started":1}
I20260926 12:41:08.886273 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:08.875423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.10.224.130:33689)
0926 12:41:08.875712 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:08.875720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:08.877088 (+ 1368us) server_negotiation.cc:407] Connection header received
0926 12:41:08.877320 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:08.877325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:08.877519 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:08.877646 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:08.878710 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.880231 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.884498 (+ 4267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.885353 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.885457 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:08.885610 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:08.885989 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 12:41:08.886109 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":104,"threads_started":1}
I20260926 12:41:08.886796 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:08.875269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33689 (local address 127.0.0.1:47902)
0926 12:41:08.876991 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:08.877004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:08.877169 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:08.877862 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:08.877868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:08.877894 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:08.878039 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:08.878046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.880426 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.880432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:08.883028 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:08.883039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.885722 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.885726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:08.885820 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:08.885844 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:41:08.886582 (+ 738us) client_negotiation.cc:240] Negotiation successful
0926 12:41:08.886668 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1662,"thread_start_us":113,"threads_started":1}
I20260926 12:41:08.886945 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:08.874411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.10.224.131:37569)
0926 12:41:08.876401 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:08.876407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:08.876432 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:41:08.876515 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:08.876521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:08.876760 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:08.879048 (+ 2288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:08.879208 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.881388 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.883236 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.884177 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.886525 (+ 2348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:08.886660 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:08.886712 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:41:08.886806 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1881,"thread_start_us":94,"threads_started":1}
I20260926 12:41:08.890369 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:08.871385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:45243 (local address 127.0.0.1:40022)
0926 12:41:08.874980 (+ 3595us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:08.875043 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:08.875191 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:08.881623 (+ 6432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:08.881628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:08.881653 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:08.881855 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:08.881863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.887330 (+ 5467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.887335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:08.888609 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:08.888621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.889935 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.889939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:08.890038 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:08.890068 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 12:41:08.890128 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 12:41:08.890219 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3388,"mutex_wait_us":4,"thread_start_us":158,"threads_started":1}
I20260926 12:41:08.890651 24308 tablet_service.cc:1467] Processing CreateTablet for tablet 156c82aee01d496782284bbfa8a71c25 (DEFAULT_TABLE table=test-table [id=2f2b50436968405e91ea4caa33d10710]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:08.891402 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:08.872307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.10.224.129:45243)
0926 12:41:08.878336 (+ 6029us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:08.878342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:08.878365 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:41:08.878444 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:08.878452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:08.878634 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:08.882017 (+ 3383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:08.883190 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.884341 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.888787 (+ 4446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:08.889778 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:08.890941 (+ 1163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:08.891086 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:08.891159 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 12:41:08.891261 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5918,"thread_start_us":128,"threads_started":1}
I20260926 12:41:08.893734 24162 tablet_service.cc:1467] Processing CreateTablet for tablet 156c82aee01d496782284bbfa8a71c25 (DEFAULT_TABLE table=test-table [id=2f2b50436968405e91ea4caa33d10710]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:08.894994 24162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c82aee01d496782284bbfa8a71c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:08.898692 24308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c82aee01d496782284bbfa8a71c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:08.898744 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 156c82aee01d496782284bbfa8a71c25 (DEFAULT_TABLE table=test-table [id=2f2b50436968405e91ea4caa33d10710]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:08.899982 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c82aee01d496782284bbfa8a71c25. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 12:41:08 dist-test-slave-m3pj krb5kdc[22912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426468, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for HTTP/127.10.224.148@KRBTEST.COM
I20260926 12:41:08.910301 24408 tablet_bootstrap.cc:492] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66: Bootstrap starting.
I20260926 12:41:08.912974 24408 tablet_bootstrap.cc:654] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:08.914052 24408 log.cc:824] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:08.917048 24408 tablet_bootstrap.cc:492] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66: No bootstrap required, opened a new log
I20260926 12:41:08.917271 24408 ts_tablet_manager.cc:1397] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:41:08.919909 24411 tablet_bootstrap.cc:492] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Bootstrap starting.
I20260926 12:41:08.921301 24408 raft_consensus.cc:348] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:08.921577 24408 raft_consensus.cc:374] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:08.921626 24408 raft_consensus.cc:729] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca0329165424eba86d8ec7261c77a66, State: Initialized, Role: FOLLOWER
I20260926 12:41:08.922398 24411 tablet_bootstrap.cc:654] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:08.922425 24408 consensus_queue.cc:260] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:08.923687 24408 ts_tablet_manager.cc:1428] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260926 12:41:08.924100 24106 heartbeater.cc:503] Master 127.10.224.190:35363 was elected leader, sending a full tablet report...
I20260926 12:41:08.924790 24409 tablet_bootstrap.cc:492] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: Bootstrap starting.
I20260926 12:41:08.927362 24409 tablet_bootstrap.cc:654] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:08.928623 24411 log.cc:824] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:08.929949 24409 log.cc:824] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:08.933001 24411 tablet_bootstrap.cc:492] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: No bootstrap required, opened a new log
I20260926 12:41:08.933471 24411 ts_tablet_manager.cc:1397] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260926 12:41:08.938926 24411 raft_consensus.cc:348] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:08.939406 24411 raft_consensus.cc:374] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:08.939723 24411 raft_consensus.cc:729] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edd7e09c08d4428bdec11181ddc08ad, State: Initialized, Role: FOLLOWER
I20260926 12:41:08.940876 24411 consensus_queue.cc:260] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:08.942764 24409 tablet_bootstrap.cc:492] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: No bootstrap required, opened a new log
I20260926 12:41:08.943046 24409 ts_tablet_manager.cc:1397] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260926 12:41:08.945084 24411 ts_tablet_manager.cc:1428] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260926 12:41:08.945552 24374 heartbeater.cc:503] Master 127.10.224.190:35363 was elected leader, sending a full tablet report...
I20260926 12:41:08.948149 24409 raft_consensus.cc:348] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:08.948560 24409 raft_consensus.cc:374] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:08.948679 24409 raft_consensus.cc:729] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7039f803b044552a810af63bab762a9, State: Initialized, Role: FOLLOWER
I20260926 12:41:08.949301 24409 consensus_queue.cc:260] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:08.950603 24409 ts_tablet_manager.cc:1428] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260926 12:41:08.951005 24240 heartbeater.cc:503] Master 127.10.224.190:35363 was elected leader, sending a full tablet report...
W20260926 12:41:08.999888 24107 tablet.cc:2369] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:41:09.012121 24241 tablet.cc:2369] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:41:09.051321 24375 tablet.cc:2369] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:41:09.056556 24412 raft_consensus.cc:482] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:09.056831 24412 raft_consensus.cc:504] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:09.058308 24412 leader_election.cc:288] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8edd7e09c08d4428bdec11181ddc08ad (127.10.224.131:37569), c7039f803b044552a810af63bab762a9 (127.10.224.130:33689)
I20260926 12:41:09.066650 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:09.059270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:52365 (local address 127.10.224.131:37569)
0926 12:41:09.059391 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:09.059396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:09.059534 (+ 138us) server_negotiation.cc:407] Connection header received
0926 12:41:09.059716 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:09.059720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:09.059771 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:09.059868 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:09.060153 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.060941 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.062738 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.063374 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.063502 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:09.063585 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:09.066400 (+ 2815us) server_negotiation.cc:299] Negotiation successful
0926 12:41:09.066506 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:41:09.067363 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:09.059137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:58149 (local address 127.10.224.130:33689)
0926 12:41:09.059273 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:09.059277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:09.059295 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:41:09.063162 (+ 3867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:09.063167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:09.063229 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:09.063307 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:09.063752 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.064598 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.065857 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.066569 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.066708 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:09.066775 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:09.067113 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 12:41:09.067206 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:41:09.067373 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:09.058739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:37569 (local address 127.10.224.129:52365)
0926 12:41:09.059435 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:09.059448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:09.059601 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:09.059874 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:09.059878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:09.059897 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:09.060047 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:09.060054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.061057 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.061061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:09.062560 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:09.062573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.063651 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.063655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:09.063786 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:09.063811 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:41:09.067106 (+ 3295us) client_negotiation.cc:240] Negotiation successful
0926 12:41:09.067194 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":118,"threads_started":1}
I20260926 12:41:09.068182 24418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:09.058649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33689 (local address 127.10.224.129:58149)
0926 12:41:09.058981 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:09.058997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:09.062977 (+ 3980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:09.063426 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:09.063429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:09.063446 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:09.063598 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:09.063605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.064718 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.064724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:09.065747 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:09.065758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.066845 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.066849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:09.066964 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:09.066993 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:41:09.067915 (+ 922us) client_negotiation.cc:240] Negotiation successful
0926 12:41:09.068016 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":145,"threads_started":1}
I20260926 12:41:09.068104 24328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c82aee01d496782284bbfa8a71c25" candidate_uuid: "9ca0329165424eba86d8ec7261c77a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd7e09c08d4428bdec11181ddc08ad" is_pre_election: true
I20260926 12:41:09.068456 24328 raft_consensus.cc:2454] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca0329165424eba86d8ec7261c77a66 in term 0.
I20260926 12:41:09.069064 23993 leader_election.cc:302] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8edd7e09c08d4428bdec11181ddc08ad, 9ca0329165424eba86d8ec7261c77a66; no voters:
I20260926 12:41:09.069057 24194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c82aee01d496782284bbfa8a71c25" candidate_uuid: "9ca0329165424eba86d8ec7261c77a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7039f803b044552a810af63bab762a9" is_pre_election: true
I20260926 12:41:09.069360 24194 raft_consensus.cc:2454] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca0329165424eba86d8ec7261c77a66 in term 0.
I20260926 12:41:09.069401 24412 raft_consensus.cc:2781] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:41:09.069476 24412 raft_consensus.cc:482] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:41:09.069516 24412 raft_consensus.cc:3037] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:09.071087 24412 raft_consensus.cc:504] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:09.071583 24412 leader_election.cc:288] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [CANDIDATE]: Term 1 election: Requested vote from peers 8edd7e09c08d4428bdec11181ddc08ad (127.10.224.131:37569), c7039f803b044552a810af63bab762a9 (127.10.224.130:33689)
I20260926 12:41:09.071926 24194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c82aee01d496782284bbfa8a71c25" candidate_uuid: "9ca0329165424eba86d8ec7261c77a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7039f803b044552a810af63bab762a9"
I20260926 12:41:09.072067 24194 raft_consensus.cc:3037] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:09.072229 24328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c82aee01d496782284bbfa8a71c25" candidate_uuid: "9ca0329165424eba86d8ec7261c77a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd7e09c08d4428bdec11181ddc08ad"
I20260926 12:41:09.072357 24328 raft_consensus.cc:3037] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:09.073459 24194 raft_consensus.cc:2454] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca0329165424eba86d8ec7261c77a66 in term 1.
I20260926 12:41:09.073628 24328 raft_consensus.cc:2454] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca0329165424eba86d8ec7261c77a66 in term 1.
I20260926 12:41:09.073954 23995 leader_election.cc:302] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca0329165424eba86d8ec7261c77a66, c7039f803b044552a810af63bab762a9; no voters:
I20260926 12:41:09.074191 24412 raft_consensus.cc:2781] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:09.074409 24412 raft_consensus.cc:686] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca0329165424eba86d8ec7261c77a66, State: Running, Role: LEADER
I20260926 12:41:09.074785 24412 consensus_queue.cc:237] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } }
I20260926 12:41:09.078733 23907 catalog_manager.cc:5495] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ca0329165424eba86d8ec7261c77a66 (127.10.224.129). New cstate: current_term: 1 leader_uuid: "9ca0329165424eba86d8ec7261c77a66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca0329165424eba86d8ec7261c77a66" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 45243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 } health_report { overall_health: UNKNOWN } } }
I20260926 12:41:09.129642 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:09.124292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.10.224.129:45243)
0926 12:41:09.124412 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:09.124417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:09.124434 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:41:09.124599 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:09.124603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:09.124674 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:09.124793 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:41:09.125187 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.125949 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.127209 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:09.127576 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:09.127681 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:09.128146 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:41:09.128800 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:41:09.128922 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:09.129395 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 12:41:09.129467 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:41:09.137869 24194 raft_consensus.cc:1261] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 1 FOLLOWER]: Refusing update from remote peer 9ca0329165424eba86d8ec7261c77a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:09.137869 24328 raft_consensus.cc:1261] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 1 FOLLOWER]: Refusing update from remote peer 9ca0329165424eba86d8ec7261c77a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:09.138813 24412 consensus_queue.cc:1035] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8edd7e09c08d4428bdec11181ddc08ad" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 37569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:09.139895 24412 consensus_queue.cc:1035] T 156c82aee01d496782284bbfa8a71c25 P 9ca0329165424eba86d8ec7261c77a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7039f803b044552a810af63bab762a9" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:09.167645 24425 mvcc.cc:205] Tried to move back new op lower bound from 7333586817578618880 to 7333586817334243328. Current Snapshot: MvccSnapshot[applied={T|T < 7333586817578618880 or (T in {7333586817578618880})}]
I20260926 12:41:09.185830 24424 mvcc.cc:205] Tried to move back new op lower bound from 7333586817578618880 to 7333586817334243328. Current Snapshot: MvccSnapshot[applied={T|T < 7333586817578618880 or (T in {7333586817578618880})}]
I20260926 12:41:09.186474 23908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:09.186653 23908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:09.190023 23908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 993d6aea81174ec5b7e7500742c08e94: Sending DeleteTablet for 3 replicas of tablet 156c82aee01d496782284bbfa8a71c25
I20260926 12:41:09.191339 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet 156c82aee01d496782284bbfa8a71c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:09 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:47902
I20260926 12:41:09.192459 24308 tablet_service.cc:1514] Processing DeleteTablet for tablet 156c82aee01d496782284bbfa8a71c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:09 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:57550
I20260926 12:41:09.194139 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 23946
I20260926 12:41:09.198927 24432 tablet_replica.cc:330] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: stopping tablet replica
I20260926 12:41:09.199249 24433 tablet_replica.cc:330] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: stopping tablet replica
I20260926 12:41:09.199790 24433 raft_consensus.cc:2229] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:09.200182 24432 raft_consensus.cc:2229] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:09.200623 24432 raft_consensus.cc:2258] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:09.201057 24433 raft_consensus.cc:2258] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:09.203454 24433 ts_tablet_manager.cc:1905] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:09.204551 24432 ts_tablet_manager.cc:1905] T 156c82aee01d496782284bbfa8a71c25 P c7039f803b044552a810af63bab762a9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 12:41:09.205266 23894 connection.cc:535] client connection to 127.10.224.129:45243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 12:41:09.205425 23894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:41:09.206000 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 24110
I20260926 12:41:09.211083 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:09.210615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:45243 (local address 127.0.0.1:40038)
0926 12:41:09.210882 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:09.210971 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260926 12:41:09.211423 23894 catalog_manager.cc:4584] TS 9ca0329165424eba86d8ec7261c77a66 (127.10.224.129:45243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 156c82aee01d496782284bbfa8a71c25: Network error: Client connection negotiation failed: client connection to 127.10.224.129:45243: connect: Connection refused (error 111)
I20260926 12:41:09.215431 24433 ts_tablet_manager.cc:1918] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:09.220566 24433 log.cc:1196] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/156c82aee01d496782284bbfa8a71c25
I20260926 12:41:09.221241 24433 ts_tablet_manager.cc:1939] T 156c82aee01d496782284bbfa8a71c25 P 8edd7e09c08d4428bdec11181ddc08ad: Deleting consensus metadata
I20260926 12:41:09.223085 23892 catalog_manager.cc:4841] TS 8edd7e09c08d4428bdec11181ddc08ad (127.10.224.131:37569): tablet 156c82aee01d496782284bbfa8a71c25 (table test-table [id=2f2b50436968405e91ea4caa33d10710]) successfully deleted
W20260926 12:41:09.224948 24401 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:41:09.224294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33689 (local address 127.0.0.1:47916)
0926 12:41:09.224477 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:09.224492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:09.224586 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:09.224845 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.130:33689: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 12:41:09.225730 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 24243
I20260926 12:41:09.234303 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 23875
I20260926 12:41:09.243490 11138 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 12:41:09.589170 11138 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 12:41:09.589293 11138 mini_ranger.cc:66] Stopping Ranger...
I20260926 12:41:09.935072 11138 mini_ranger.cc:68] Stopped Ranger
2026-09-26 12:41:09.935 UTC [22942] LOG: received smart shutdown request
2026-09-26 12:41:09.940 UTC [22942] LOG: background worker "logical replication launcher" (PID 22951) exited with exit code 1
2026-09-26 12:41:09.940 UTC [22946] LOG: shutting down
2026-09-26 12:41:10.141 UTC [22942] LOG: database system is shut down
2026-09-26T12:41:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51148 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:41:10 dist-test-slave-m3pj krb5kdc[24445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:41:10 dist-test-slave-m3pj krb5kdc[24445](info): set up 2 sockets
Sep 26 12:41:10 dist-test-slave-m3pj krb5kdc[24445](info): commencing operation
krb5kdc: starting...
W20260926 12:41:11.332654 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:41:11 dist-test-slave-m3pj krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:41:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:41:11.466763 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:43553
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:43553,127.10.224.189:40489,127.10.224.188:41649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:11.580312 24458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:11.580627 24458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:11.580672 24458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:11.584867 24458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:11.584944 24458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:11.584964 24458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:11.584981 24458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:11.584997 24458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:11.590540 24458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:43553,127.10.224.189:40489,127.10.224.188:41649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:43553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:11.591703 24458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:11.592720 24458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:11.601354 24464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:11.601354 24463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:11.601354 24466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:11.601534 24458 server_base.cc:1034] running on GCE node
I20260926 12:41:11.602234 24458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:11.603391 24458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:11.604560 24458 hybrid_clock.cc:648] HybridClock initialized: now 1790426471604533 us; error 59 us; skew 500 ppm
Sep 26 12:41:11 dist-test-slave-m3pj krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:11.607403 24458 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:11.608355 24458 webserver.cc:460] Webserver started at http://127.10.224.190:42711/ using document root <none> and password file <none>
I20260926 12:41:11.608903 24458 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:11.608961 24458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:11.609125 24458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:11.611150 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "f8d8a416a23348e98842b2d2ee9377d3"
format_stamp: "Formatted at 2026-09-26 12:41:11 on dist-test-slave-m3pj"
server_key: "79af4bcda6f4b76eff7d9249a713f3c7"
server_key_iv: "1f8a177943401a4b10e9f0c96aeb1c70"
server_key_version: "encryptionkey@0"
I20260926 12:41:11.611769 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "f8d8a416a23348e98842b2d2ee9377d3"
format_stamp: "Formatted at 2026-09-26 12:41:11 on dist-test-slave-m3pj"
server_key: "79af4bcda6f4b76eff7d9249a713f3c7"
server_key_iv: "1f8a177943401a4b10e9f0c96aeb1c70"
server_key_version: "encryptionkey@0"
I20260926 12:41:11.615408 24458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:11.617944 24473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:11.619212 24458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:41:11.619359 24458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "f8d8a416a23348e98842b2d2ee9377d3"
format_stamp: "Formatted at 2026-09-26 12:41:11 on dist-test-slave-m3pj"
server_key: "79af4bcda6f4b76eff7d9249a713f3c7"
server_key_iv: "1f8a177943401a4b10e9f0c96aeb1c70"
server_key_version: "encryptionkey@0"
I20260926 12:41:11.619473 24458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:11.633935 24458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:11.637091 24458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:11.637313 24458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:11.645203 24458 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:43553
I20260926 12:41:11.645515 24525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:43553 every 8 connection(s)
I20260926 12:41:11.646260 24458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:11.649185 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:11.653169 24526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:41:11.653841 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 24458
I20260926 12:41:11.653993 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:41:11.654281 11138 external_mini_cluster.cc:1428] Setting key 538561e78cde9d44d557b8638d39d9ed
WARNING: no policy specified for kudu/127.10.224.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.189@KRBTEST.COM" created.
Sep 26 12:41:11 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:11.672374 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.656066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.10.224.190:43553)
0926 12:41:11.656742 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:11.656756 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:11.656811 (+ 55us) server_negotiation.cc:407] Connection header received
0926 12:41:11.658297 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:11.658302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:11.658518 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:11.658694 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:11.659914 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.660764 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.662334 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.662696 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.662816 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:11.667156 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:11.667166 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:11.667185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:11.667219 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:11.669029 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:11.669617 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:11.669622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:11.669623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:11.669725 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:11.670119 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:11.670122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:11.670125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:11.670536 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:11.670640 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:11.671146 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 12:41:11.671430 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":515,"thread_start_us":172,"threads_started":1}
I20260926 12:41:11.672380 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.656066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43553 (local address 127.0.0.1:55970)
0926 12:41:11.656627 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.656681 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:11.657895 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:11.658758 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:11.658767 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:11.659275 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:11.659711 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:11.659747 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.660892 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.660898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:11.662194 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:11.662210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.662895 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.662901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:11.663044 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:11.663815 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:11.663832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:11.666983 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:11.669149 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:11.669162 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:11.669176 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:11.669503 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:11.669819 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:11.669825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:11.669830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:11.669985 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:11.670660 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:11.670669 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:11.670869 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 12:41:11.671081 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 12:41:11.671559 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":95,"thread_start_us":135,"threads_started":1}
I20260926 12:41:11.674764 24526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:41:11.675498 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.675233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35166)
0926 12:41:11.675363 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.675437 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 12:41:11.675757 24475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
I20260926 12:41:11.676700 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.676417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35172)
0926 12:41:11.676522 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.676604 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:41:11.676851 24526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:40489: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab.
Entry for principal kudu/127.10.224.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab.
I20260926 12:41:11.683830 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.189:40489
--webserver_interface=127.10.224.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab
--principal=kudu/127.10.224.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:43553,127.10.224.189:40489,127.10.224.188:41649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:11.726151 24526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } attempt: 1
I20260926 12:41:11.727056 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.726712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35186)
0926 12:41:11.726885 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.726960 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 12:41:11.727808 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.727601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35196)
0926 12:41:11.727700 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.727750 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 12:41:11.727937 24526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:40489: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
I20260926 12:41:11.796118 24526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } attempt: 2
W20260926 12:41:11.796041 24533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:11.796473 24533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:11.796586 24533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:41:11.797194 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.796818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35206)
0926 12:41:11.796986 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.797087 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 12:41:11.797952 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.797719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35220)
0926 12:41:11.797834 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.797886 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 12:41:11.798120 24526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:40489: Network error: Client connection negotiation failed: client connection to 127.10.224.189:40489: connect: Connection refused (error 111)
W20260926 12:41:11.801524 24533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:11.801618 24533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:11.801664 24533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:11.801726 24533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:11.801764 24533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:11.807423 24533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:43553,127.10.224.189:40489,127.10.224.188:41649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab
--principal=kudu/127.10.224.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.189:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.224.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:11.808574 24533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:11.809581 24533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:11.818957 24539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:11.818959 24538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:11.818957 24541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:11.819304 24533 server_base.cc:1034] running on GCE node
I20260926 12:41:11.820048 24533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:11.821265 24533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:11.822472 24533 hybrid_clock.cc:648] HybridClock initialized: now 1790426471822424 us; error 62 us; skew 500 ppm
Sep 26 12:41:11 dist-test-slave-m3pj krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:11.825752 24533 init.cc:377] Logged in from keytab as kudu/127.10.224.189@KRBTEST.COM (short username kudu)
I20260926 12:41:11.826663 24533 webserver.cc:460] Webserver started at http://127.10.224.189:32909/ using document root <none> and password file <none>
I20260926 12:41:11.827203 24533 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:11.827265 24533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:11.827437 24533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:11.829587 24533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/data/instance:
uuid: "e2ac0ab4a1d447608a3dc23a55517a03"
format_stamp: "Formatted at 2026-09-26 12:41:11 on dist-test-slave-m3pj"
server_key: "1d0587f3abf385b89b86ff9481175be8"
server_key_iv: "e844996146fd2dc0beb900d446fd418f"
server_key_version: "encryptionkey@0"
I20260926 12:41:11.830119 24533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/wal/instance:
uuid: "e2ac0ab4a1d447608a3dc23a55517a03"
format_stamp: "Formatted at 2026-09-26 12:41:11 on dist-test-slave-m3pj"
server_key: "1d0587f3abf385b89b86ff9481175be8"
server_key_iv: "e844996146fd2dc0beb900d446fd418f"
server_key_version: "encryptionkey@0"
I20260926 12:41:11.833906 24533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:41:11.836519 24548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:11.837791 24533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:11.837941 24533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/wal
uuid: "e2ac0ab4a1d447608a3dc23a55517a03"
format_stamp: "Formatted at 2026-09-26 12:41:11 on dist-test-slave-m3pj"
server_key: "1d0587f3abf385b89b86ff9481175be8"
server_key_iv: "e844996146fd2dc0beb900d446fd418f"
server_key_version: "encryptionkey@0"
I20260926 12:41:11.838061 24533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:11.865087 24533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:11.868657 24533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:11.868929 24533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:11.876474 24533 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.189:40489
I20260926 12:41:11.876753 24600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.189:40489 every 8 connection(s)
I20260926 12:41:11.877584 24533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
I20260926 12:41:11.880533 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 24533
I20260926 12:41:11.880647 24601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:11.880682 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-1/wal/instance
I20260926 12:41:11.880998 11138 external_mini_cluster.cc:1428] Setting key 372fadd981d9af92b1acd5beab3d71c2
I20260926 12:41:11.885999 24601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:41:11.889331 24526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } attempt: 3
WARNING: no policy specified for kudu/127.10.224.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.188@KRBTEST.COM" created.
Sep 26 12:41:11 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
Sep 26 12:41:11 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:11.901379 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.889759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35226)
0926 12:41:11.890040 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.890053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:11.890134 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:11.891185 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:11.891188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:11.891349 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:11.891438 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:11.891443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.893065 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.893071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:11.893673 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:11.893684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.894403 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.894409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:11.894596 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:11.895304 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:11.895324 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:11.896818 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:11.899309 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:11.899312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:11.899315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:11.899512 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:11.899928 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:11.899931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:11.899933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:11.900040 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:11.900795 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:11.900798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:11.900905 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 12:41:11.901024 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 12:41:11.901190 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":24,"spinlock_wait_cycles":768}
I20260926 12:41:11.903126 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.888223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.10.224.190:43553)
0926 12:41:11.888381 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:11.888385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:11.889613 (+ 1228us) server_negotiation.cc:407] Connection header received
0926 12:41:11.890968 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:11.890970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:11.891015 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:11.891124 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:11.892113 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.892657 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.894132 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.894478 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.894599 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:11.897845 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:11.897852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:11.897856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:11.898179 (+ 323us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:11.900310 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:11.900920 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:11.900923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:11.900926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:11.901014 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:11.901324 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:11.901327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:11.901328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:11.901535 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:11.901653 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:11.902907 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0926 12:41:11.903009 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":60,"spinlock_wait_cycles":1280}
I20260926 12:41:11.903366 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.888780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43553 (local address 127.0.0.1:55976)
0926 12:41:11.889444 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.889504 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:11.890752 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:11.891299 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:11.891304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:11.891752 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:11.891973 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:11.891986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.892831 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.892834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:11.893995 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:11.894009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.894696 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.894700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:11.894823 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:11.895564 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:11.895610 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:11.897681 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:11.900425 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:11.900436 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:11.900445 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:11.900781 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:11.901114 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:11.901119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:11.901121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:11.901214 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:11.901706 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:11.901713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:11.901872 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 12:41:11.902014 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 12:41:11.902311 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":111,"thread_start_us":162,"threads_started":1}
I20260926 12:41:11.903366 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.890055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.10.224.189:40489)
0926 12:41:11.890419 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:11.890430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:11.890474 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:41:11.890752 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:11.890775 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:11.891020 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:11.891223 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:11.891719 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.892901 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.893798 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.894266 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.894436 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:11.896969 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:11.896976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:11.896990 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:11.897020 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:11.899155 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:11.899653 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:11.899661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:11.899667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:11.899811 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:11.900155 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:11.900159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:11.900163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:11.900684 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:11.900830 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:11.901896 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0926 12:41:11.902260 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":179,"threads_started":1}
I20260926 12:41:11.905421 24601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:41:11.905949 24526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:41:11.906595 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.906351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53370)
0926 12:41:11.906466 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.906518 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 12:41:11.907226 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.906991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53382)
0926 12:41:11.907123 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.907167 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 12:41:11.907370 24526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:41649: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Sep 26 12:41:11 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab.
Entry for principal kudu/127.10.224.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab.
I20260926 12:41:11.913007 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.188:41649
--webserver_interface=127.10.224.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab
--principal=kudu/127.10.224.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:43553,127.10.224.189:40489,127.10.224.188:41649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:11.914498 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.906153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.10.224.189:40489)
0926 12:41:11.906256 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:11.906262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:11.906279 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:41:11.906312 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:11.906315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:11.906348 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:11.906438 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:11.906916 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.907665 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.908547 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.908796 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.908907 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:11.910989 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:11.910992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:11.910994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:11.911014 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:11.912761 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:11.913249 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:11.913255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:11.913257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:11.913335 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:11.913610 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:11.913613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:11.913615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:11.913807 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:11.913912 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:11.914151 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 12:41:11.914307 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":46}
I20260926 12:41:11.914498 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.906002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35232)
0926 12:41:11.906136 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.906150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:11.906237 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:11.906449 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:11.906451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:11.906677 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:11.906799 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:11.906806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.907796 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.907799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:11.908440 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:11.908448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:11.908912 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:11.908915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:11.909048 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:11.909509 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:11.909528 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:11.910856 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:11.912892 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:11.912894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:11.912896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:11.913079 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:11.913429 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:11.913434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:11.913436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:11.913518 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:11.913901 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:11.913904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:11.913984 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 12:41:11.914099 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 12:41:11.914251 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260926 12:41:11.915925 24601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:41:11.916761 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.916488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53392)
0926 12:41:11.916624 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.916700 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 12:41:11.917339 24550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
I20260926 12:41:11.918308 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.918082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53400)
0926 12:41:11.918202 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.918255 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:41:11.918470 24601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:41649: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
I20260926 12:41:11.938565 24526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } attempt: 1
I20260926 12:41:11.939613 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.939225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53402)
0926 12:41:11.939402 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.939477 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 12:41:11.940423 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.940182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53414)
0926 12:41:11.940301 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.940355 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:41:11.940567 24526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:41649: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
I20260926 12:41:11.967742 24601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } attempt: 1
I20260926 12:41:11.968734 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.968402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53424)
0926 12:41:11.968582 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.968655 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 12:41:11.969728 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:11.969390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53438)
0926 12:41:11.969562 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:11.969635 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260926 12:41:11.969944 24601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:41649: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
I20260926 12:41:12.015723 24526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } attempt: 2
I20260926 12:41:12.016630 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.016281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53444)
0926 12:41:12.016457 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.016532 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 12:41:12.017395 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.017164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53450)
0926 12:41:12.017276 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.017327 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:41:12.017534 24526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:41649: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
W20260926 12:41:12.033136 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:12.033427 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:12.033506 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:12.037782 24607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:12.037865 24607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:12.037886 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:12.037935 24607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:12.037967 24607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:12.038141 24601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } attempt: 2
I20260926 12:41:12.038992 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.038688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53466)
0926 12:41:12.038835 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.038912 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 12:41:12.039975 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.039752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53468)
0926 12:41:12.039863 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.039915 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 12:41:12.040132 24601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:41649: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
I20260926 12:41:12.044370 24607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:43553,127.10.224.189:40489,127.10.224.188:41649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab
--principal=kudu/127.10.224.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.188:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.224.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:12.045578 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:12.046685 24607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:12.055686 24613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:12.055686 24614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:12.055686 24616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:12.056265 24607 server_base.cc:1034] running on GCE node
I20260926 12:41:12.056862 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:12.058154 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:12.059386 24607 hybrid_clock.cc:648] HybridClock initialized: now 1790426472059346 us; error 51 us; skew 500 ppm
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:12.062534 24607 init.cc:377] Logged in from keytab as kudu/127.10.224.188@KRBTEST.COM (short username kudu)
I20260926 12:41:12.063661 24607 webserver.cc:460] Webserver started at http://127.10.224.188:45895/ using document root <none> and password file <none>
I20260926 12:41:12.064353 24607 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:12.064446 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:12.064662 24607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:12.066792 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/data/instance:
uuid: "114bba2b0a2449ec9283f4ae3d212a7c"
format_stamp: "Formatted at 2026-09-26 12:41:12 on dist-test-slave-m3pj"
server_key: "43c4d8a3e353c36ff96810353fc7b83d"
server_key_iv: "7be4e35b501b802e45275a5232ba4f9c"
server_key_version: "encryptionkey@0"
I20260926 12:41:12.067338 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/wal/instance:
uuid: "114bba2b0a2449ec9283f4ae3d212a7c"
format_stamp: "Formatted at 2026-09-26 12:41:12 on dist-test-slave-m3pj"
server_key: "43c4d8a3e353c36ff96810353fc7b83d"
server_key_iv: "7be4e35b501b802e45275a5232ba4f9c"
server_key_version: "encryptionkey@0"
I20260926 12:41:12.071420 24607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 12:41:12.074213 24623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:12.075702 24607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:12.075845 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/wal
uuid: "114bba2b0a2449ec9283f4ae3d212a7c"
format_stamp: "Formatted at 2026-09-26 12:41:12 on dist-test-slave-m3pj"
server_key: "43c4d8a3e353c36ff96810353fc7b83d"
server_key_iv: "7be4e35b501b802e45275a5232ba4f9c"
server_key_version: "encryptionkey@0"
I20260926 12:41:12.075978 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:12.107890 24607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:12.111236 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:12.111469 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:12.116703 24526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } attempt: 3
I20260926 12:41:12.117681 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.117290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53470)
0926 12:41:12.117485 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.117569 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 12:41:12.118494 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.118247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53480)
0926 12:41:12.118368 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.118425 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:41:12.118680 24526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:41649: Network error: Client connection negotiation failed: client connection to 127.10.224.188:41649: connect: Connection refused (error 111)
I20260926 12:41:12.119385 24607 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.188:41649
I20260926 12:41:12.119711 24675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.188:41649 every 8 connection(s)
I20260926 12:41:12.120559 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
I20260926 12:41:12.120975 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 24607
I20260926 12:41:12.121119 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790426298078858-11138-0/minicluster-data/master-2/wal/instance
I20260926 12:41:12.121692 11138 external_mini_cluster.cc:1428] Setting key 69eef289c979e945d3423a1f15ed9217
I20260926 12:41:12.124554 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:12.128659 24676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:41:12.131305 24601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } attempt: 3
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:41:12.138262 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.123297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.10.224.188:41649)
0926 12:41:12.123816 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.123840 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.123897 (+ 57us) server_negotiation.cc:407] Connection header received
0926 12:41:12.124830 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.124850 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.125243 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.125620 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.126462 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.127466 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.128358 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.128849 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.129026 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.131859 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.131867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.131880 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.131918 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.134422 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.134995 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.135001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.135008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.135145 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.135481 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.135485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.135487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.136119 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.136376 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.136740 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.137084 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":184,"threads_started":1}
I20260926 12:41:12.141924 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.130661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.10.224.190:43553)
0926 12:41:12.130820 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.130825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.131161 (+ 336us) server_negotiation.cc:407] Connection header received
0926 12:41:12.131331 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.131334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.131398 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.131474 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.132670 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.133186 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.134404 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.134729 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.134853 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.138153 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.138157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.138161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.138192 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.139855 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.140416 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.140420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.140422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.140515 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.140803 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.140807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.140809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.141059 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.141165 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.141601 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.141750 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 12:41:12.142625 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.130580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43553 (local address 127.0.0.1:55978)
0926 12:41:12.131011 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.131066 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:12.131194 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:12.131724 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:12.131730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:12.132286 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:12.132479 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.132489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.133310 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.133314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.134262 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.134272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.134979 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.134982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.135165 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.135732 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:12.135745 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:12.137957 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:12.139988 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.139996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.140001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.140292 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.140608 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.140614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.140619 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.140706 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.141159 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:12.141165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:12.141292 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 12:41:12.142173 (+ 881us) client_negotiation.cc:240] Negotiation successful
0926 12:41:12.142381 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":21,"thread_start_us":168,"threads_started":1}
I20260926 12:41:12.142844 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.131801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53496)
0926 12:41:12.131955 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.131972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:12.132059 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:12.132572 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:12.132575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:12.132747 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:12.132830 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.132835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.133884 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.133887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.134675 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.134687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.135379 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.135382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.135768 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.136230 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:12.136266 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:12.137842 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:12.141092 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.141105 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.141109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.141380 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.141754 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.141757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.141759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.141854 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.142313 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:12.142316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:12.142426 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 12:41:12.142535 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 12:41:12.142683 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 12:41:12.142956 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.132036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.10.224.188:41649)
0926 12:41:12.132361 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.132368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.132379 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:41:12.132405 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.132407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.132463 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.132551 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.132943 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.133752 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.134812 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.135203 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.135359 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.137983 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.137987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.137990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.138013 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.140932 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.141538 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.141541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.141543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.141633 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.141961 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.141967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.141971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.142217 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.142326 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.142641 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.142757 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260926 12:41:12.143769 24676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:41:12.146530 24601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03: Bootstrap starting.
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
I20260926 12:41:12.149541 24601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:12.150434 24601 log.cc:824] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:12.152184 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.141523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.10.224.190:43553)
0926 12:41:12.141807 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.141825 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.141835 (+ 10us) server_negotiation.cc:407] Connection header received
0926 12:41:12.141862 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.141864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.141898 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.142856 (+ 958us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.142961 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.143783 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.144837 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.145099 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.145560 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.147949 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.147952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.147954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.147977 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.149712 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.150275 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.150278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.150279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.150357 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.151003 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.151007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.151009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.151257 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.151511 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.151757 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.151934 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
I20260926 12:41:12.153083 24601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03: No bootstrap required, opened a new log
I20260926 12:41:12.154213 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.144369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.10.224.189:40489)
0926 12:41:12.144473 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.144477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.144509 (+ 32us) server_negotiation.cc:407] Connection header received
0926 12:41:12.144614 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.144616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.144657 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.144741 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.145166 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.145856 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.146847 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.147200 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.147384 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.150053 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.150057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.150060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.150084 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.152119 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.152876 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.152882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.152883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.152970 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.153322 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.153327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.153330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.153591 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.153707 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.154000 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.154112 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:41:12.154647 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.144291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:40489 (local address 127.0.0.1:35240)
0926 12:41:12.144450 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.144464 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:12.144547 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:12.144769 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:12.144772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:12.144934 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:12.145046 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.145052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.145973 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.145976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.146723 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.146734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.147506 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.147513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.147658 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.148132 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:12.148147 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:12.149906 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:12.152241 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.152248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.152251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.152512 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.153063 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.153066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.153069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.153183 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.153698 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:12.153701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:12.153806 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 12:41:12.154352 (+ 546us) client_negotiation.cc:240] Negotiation successful
0926 12:41:12.154506 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 12:41:12.155753 24676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:41:12.157897 24601 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
I20260926 12:41:12.158169 24601 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:12.158248 24601 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ac0ab4a1d447608a3dc23a55517a03, State: Initialized, Role: FOLLOWER
I20260926 12:41:12.159072 24601 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
I20260926 12:41:12.159706 24685 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.159956 24685 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:41:12.160727 24601 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:41:12.165885 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.156326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53512)
0926 12:41:12.156462 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.156476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:12.156562 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:12.156770 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:12.156773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:12.156971 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:12.157034 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.157039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.157750 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.157754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.158330 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.158339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.159408 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.159415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.159506 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.160026 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:12.160040 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:12.161984 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:12.164161 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.164164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.164166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.164387 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.164732 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.164736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.164738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.164838 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.165349 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:12.165355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:12.165446 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 12:41:12.165544 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 12:41:12.165710 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 12:41:12.166461 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.156356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.10.224.188:41649)
0926 12:41:12.156452 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.156456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.156541 (+ 85us) server_negotiation.cc:407] Connection header received
0926 12:41:12.156631 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.156635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.156669 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.156760 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.157140 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.157638 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.158456 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.158773 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.158882 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.162130 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.162134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.162136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.162226 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.164041 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.164499 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.164503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.164505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.164607 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.164935 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.164938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.164939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.165228 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.165316 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.166191 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.166313 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 12:41:12.165026 24533 master_runner.cc:186] Error getting master registration for 127.10.224.190:43553: OK, instance_id { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" instance_seqno: 1790426471643092 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:41:12.169456 24676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c: Bootstrap starting.
I20260926 12:41:12.168414 24533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 12:41:12.167136 24696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 12:41:12.171768 24676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:12.171872 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.153783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.10.224.189:40489)
0926 12:41:12.154107 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.154117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.154132 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:41:12.154167 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.154171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.154215 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.155393 (+ 1178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.155686 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.156980 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.158157 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.158546 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.158659 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.161684 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.161690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.161693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.161724 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.164783 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.165315 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.165318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.165320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.165423 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.165745 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.165747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.165749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.165976 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.167185 (+ 1209us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.167263 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.167361 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":122,"threads_started":1}
W20260926 12:41:12.171734 24696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:41:12.172645 24676 log.cc:824] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:12.174868 24676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c: No bootstrap required, opened a new log
I20260926 12:41:12.175766 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:41:12.175801 24685 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:12.175944 24685 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
I20260926 12:41:12.176952 24685 leader_election.cc:288] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8d8a416a23348e98842b2d2ee9377d3 (127.10.224.190:43553), 114bba2b0a2449ec9283f4ae3d212a7c (127.10.224.188:41649)
I20260926 12:41:12.180017 24676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
I20260926 12:41:12.180325 24676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:12.180385 24676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 114bba2b0a2449ec9283f4ae3d212a7c, State: Initialized, Role: FOLLOWER
I20260926 12:41:12.180976 24676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
I20260926 12:41:12.181576 24706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.181792 24706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:41:12.182490 24676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:12.182523 24650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" is_pre_election: true
I20260926 12:41:12.182828 24650 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ac0ab4a1d447608a3dc23a55517a03 in term 0.
I20260926 12:41:12.183301 24550 leader_election.cc:302] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114bba2b0a2449ec9283f4ae3d212a7c, e2ac0ab4a1d447608a3dc23a55517a03; no voters:
I20260926 12:41:12.183683 24685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:41:12.183775 24685 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:41:12.183821 24685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:12.185155 24685 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
I20260926 12:41:12.185564 24685 leader_election.cc:288] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [CANDIDATE]: Term 1 election: Requested vote from peers f8d8a416a23348e98842b2d2ee9377d3 (127.10.224.190:43553), 114bba2b0a2449ec9283f4ae3d212a7c (127.10.224.188:41649)
I20260926 12:41:12.186036 24650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114bba2b0a2449ec9283f4ae3d212a7c"
I20260926 12:41:12.186156 24650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:12.187501 24650 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ac0ab4a1d447608a3dc23a55517a03 in term 1.
I20260926 12:41:12.187868 24550 leader_election.cc:302] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 114bba2b0a2449ec9283f4ae3d212a7c, e2ac0ab4a1d447608a3dc23a55517a03; no voters:
I20260926 12:41:12.188084 24685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:12.188282 24685 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [term 1 LEADER]: Becoming Leader. State: Replica: e2ac0ab4a1d447608a3dc23a55517a03, State: Running, Role: LEADER
I20260926 12:41:12.188621 24685 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
W20260926 12:41:12.189661 24717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:41:12.189746 24717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:41:12.186561 24607 master_runner.cc:186] Error getting master registration for 127.10.224.190:43553: OK, instance_id { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" instance_seqno: 1790426471643092 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:41:12.189916 24607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:41:12.190485 24685 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ac0ab4a1d447608a3dc23a55517a03. Latest consensus state: current_term: 1 leader_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.190589 24685 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:12.190929 24721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:12.191864 24721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:12.195950 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.178168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.10.224.190:43553)
0926 12:41:12.178305 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.178310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.178985 (+ 675us) server_negotiation.cc:407] Connection header received
0926 12:41:12.179368 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.179371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.179444 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.179500 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.181623 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.182427 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.184997 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.185410 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.185503 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.190514 (+ 5011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.190520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.190523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.190557 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.192615 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.193681 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.193699 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.193701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.193805 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.194417 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.194422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.194424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.194689 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.194822 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.195603 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.195759 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:41:12.198787 24650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [term 1 FOLLOWER]: Refusing update from remote peer e2ac0ab4a1d447608a3dc23a55517a03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:12.199487 24685 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:12.203753 24706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ac0ab4a1d447608a3dc23a55517a03. Latest consensus state: current_term: 1 leader_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.203905 24706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:41:12.209187 24719 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.209337 24719 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:12.210592 24706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.210726 24706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:41:12.211091 24722 mvcc.cc:205] Tried to move back new op lower bound from 7333586830118174720 to 7333586830088019968. Current Snapshot: MvccSnapshot[applied={T|T < 7333586830118174720}]
I20260926 12:41:12.211583 24723 mvcc.cc:205] Tried to move back new op lower bound from 7333586830118174720 to 7333586830088019968. Current Snapshot: MvccSnapshot[applied={T|T < 7333586830118174720}]
I20260926 12:41:12.215211 24721 catalog_manager.cc:1347] Generated new cluster ID: eac852a9f36e4b139ab4750713c43521
I20260926 12:41:12.215291 24721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:41:12.215409 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.179679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.10.224.188:41649)
0926 12:41:12.179817 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.179822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.199782 (+ 19960us) server_negotiation.cc:407] Connection header received
0926 12:41:12.199963 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.199967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.200022 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.200114 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.201216 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.202181 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.203574 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.203965 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.204078 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.206299 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.206303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.206307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.206335 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.208262 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.209643 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.209647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.209690 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.209924 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.211026 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.211030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.211074 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.212171 (+ 1097us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.213362 (+ 1191us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.213810 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.214464 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:41:12.219944 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.178899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.10.224.189:40489)
0926 12:41:12.179022 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.179026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.198778 (+ 19752us) server_negotiation.cc:407] Connection header received
0926 12:41:12.199327 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.199333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.199391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.199451 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.206035 (+ 6584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.207510 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.208822 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.209269 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.209359 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.211238 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.211242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.211245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.211274 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.213272 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.213809 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.213813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.213816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.213907 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.214240 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.214245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.214249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.214492 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.214617 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.215021 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.215162 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:41:12.232872 24721 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:41:12.234242 24721 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:41:12.243016 24721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03: Generated new TSK 0
I20260926 12:41:12.243991 24721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:41:12.282915 24526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } attempt: 4
Sep 26 12:41:12 dist-test-slave-m3pj krb5kdc[24445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:41:12.294034 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.283772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:41649 (local address 127.0.0.1:53534)
0926 12:41:12.284177 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:12.284191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:12.284286 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:12.284914 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:12.284918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:12.285144 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:12.285250 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.285256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.286379 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.286383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.287230 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:12.287240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.287925 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.287929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:12.288022 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.288643 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:12.288659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:12.290290 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:12.292418 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.292421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.292423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.292668 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.292969 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:12.292972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:12.292974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:12.293055 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:12.293507 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:12.293510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:12.293608 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 12:41:12.293706 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 12:41:12.293851 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260926 12:41:12.294605 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:12.284469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.10.224.188:41649)
0926 12:41:12.284602 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:12.284607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:12.284620 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:41:12.284666 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:12.284669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:12.284714 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:12.284788 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:12.285385 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.286242 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.287367 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:12.287747 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:12.287836 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:12.290460 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:12.290464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:12.290466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:12.290495 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:12.292287 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.292775 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.292779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.292781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.292871 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:12.293163 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:12.293166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:12.293168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:12.293382 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:12.293466 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:12.294327 (+ 861us) server_negotiation.cc:299] Negotiation successful
0926 12:41:12.294450 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:41:12.297947 24526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3: Bootstrap starting.
I20260926 12:41:12.300976 24526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:12.302021 24526 log.cc:824] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:12.304428 24526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3: No bootstrap required, opened a new log
I20260926 12:41:12.308682 24526 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
I20260926 12:41:12.308945 24526 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:12.309005 24526 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d8a416a23348e98842b2d2ee9377d3, State: Initialized, Role: FOLLOWER
I20260926 12:41:12.309633 24526 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } }
I20260926 12:41:12.311151 24526 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:12.311409 24498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:12.313107 24498 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [term 1 FOLLOWER]: Refusing update from remote peer e2ac0ab4a1d447608a3dc23a55517a03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:12.313586 24499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d8a416a23348e98842b2d2ee9377d3"
I20260926 12:41:12.313896 24685 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:12.315016 24499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ac0ab4a1d447608a3dc23a55517a03 in term 1.
I20260926 12:41:12.315325 24500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d8a416a23348e98842b2d2ee9377d3" is_pre_election: true
I20260926 12:41:12.315964 24731 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.316169 24731 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:41:12.320322 24731 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ac0ab4a1d447608a3dc23a55517a03. Latest consensus state: current_term: 1 leader_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.321585 24742 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.321702 24742 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:41:12.321159 24731 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:41:12.323271 24743 mvcc.cc:205] Tried to move back new op lower bound from 7333586830290993152 to 7333586830088019968. Current Snapshot: MvccSnapshot[applied={T|T < 7333586830118174720}]
W20260926 12:41:12.330075 24745 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:41:12.330186 24745 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:41:12.861225 24685 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8d8a416a23348e98842b2d2ee9377d3" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43553 } } peers { permanent_uuid: "e2ac0ab4a1d447608a3dc23a55517a03" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 40489 } } peers { permanent_uuid: "114bba2b0a2449ec9283f4ae3d212a7c" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 41649 } } }
I20260926 12:41:12.861477 24685 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ac0ab4a1d447608a3dc23a55517a03 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:13.191112 24717 catalog_manager.cc:1259] Loaded cluster ID: eac852a9f36e4b139ab4750713c43521
I20260926 12:41:13.191218 24717 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c: loading cluster ID for follower catalog manager: success
I20260926 12:41:13.192961 24717 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c: acquiring CA information for follower catalog manager: success
I20260926 12:41:13.194118 24717 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 114bba2b0a2449ec9283f4ae3d212a7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:41:13.331362 24745 catalog_manager.cc:1259] Loaded cluster ID: eac852a9f36e4b139ab4750713c43521
I20260926 12:41:13.331487 24745 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3: loading cluster ID for follower catalog manager: success
I20260926 12:41:13.333210 24745 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3: acquiring CA information for follower catalog manager: success
I20260926 12:41:13.334457 24745 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8d8a416a23348e98842b2d2ee9377d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:41:14.317998 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 24458
I20260926 12:41:14.324724 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 24533
I20260926 12:41:14.330812 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 24607
2026-09-26T12:41:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4063 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:41:14 dist-test-slave-m3pj krb5kdc[24755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:41:14 dist-test-slave-m3pj krb5kdc[24755](info): set up 2 sockets
Sep 26 12:41:14 dist-test-slave-m3pj krb5kdc[24755](info): commencing operation
krb5kdc: starting...
W20260926 12:41:15.376415 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:41:15 dist-test-slave-m3pj krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:41:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:41:15.523444 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35119
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:35119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:15.641371 24768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:15.641675 24768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:15.641718 24768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:15.646165 24768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:15.646250 24768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:15.646276 24768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:15.646301 24768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:15.646346 24768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:15.652312 24768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:35119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:35119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:15.653575 24768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:15.654770 24768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:15.664096 24776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:15.664130 24773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:15.664433 24768 server_base.cc:1034] running on GCE node
W20260926 12:41:15.664129 24774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:15.665264 24768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:15.666584 24768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:15.667771 24768 hybrid_clock.cc:648] HybridClock initialized: now 1790426475667751 us; error 42 us; skew 500 ppm
Sep 26 12:41:15 dist-test-slave-m3pj krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426475, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:15.671038 24768 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:15.672264 24768 webserver.cc:460] Webserver started at http://127.10.224.190:45547/ using document root <none> and password file <none>
I20260926 12:41:15.672976 24768 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:15.673079 24768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:15.673364 24768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:15.675836 24768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "f9743e123299452da910132e9efccc28"
format_stamp: "Formatted at 2026-09-26 12:41:15 on dist-test-slave-m3pj"
server_key: "56dc93b77c8c843d5b08d1325fc0598e"
server_key_iv: "23ec0e981fcb6fd22cc387762ef4bc9e"
server_key_version: "encryptionkey@0"
I20260926 12:41:15.676522 24768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "f9743e123299452da910132e9efccc28"
format_stamp: "Formatted at 2026-09-26 12:41:15 on dist-test-slave-m3pj"
server_key: "56dc93b77c8c843d5b08d1325fc0598e"
server_key_iv: "23ec0e981fcb6fd22cc387762ef4bc9e"
server_key_version: "encryptionkey@0"
I20260926 12:41:15.680989 24768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:41:15.684063 24783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:15.685640 24768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:41:15.685824 24768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "f9743e123299452da910132e9efccc28"
format_stamp: "Formatted at 2026-09-26 12:41:15 on dist-test-slave-m3pj"
server_key: "56dc93b77c8c843d5b08d1325fc0598e"
server_key_iv: "23ec0e981fcb6fd22cc387762ef4bc9e"
server_key_version: "encryptionkey@0"
I20260926 12:41:15.685985 24768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:15.699962 24768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:15.704025 24768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:15.704272 24768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:15.711769 24768 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:35119
I20260926 12:41:15.711793 24835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:35119 every 8 connection(s)
I20260926 12:41:15.712836 24768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:15.715865 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:15.720880 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 24768
I20260926 12:41:15.721025 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:41:15.721354 11138 external_mini_cluster.cc:1428] Setting key 7cf6b99d56a6ae177122fb1875ea73a4
I20260926 12:41:15.722437 24836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28: Bootstrap starting.
I20260926 12:41:15.725055 24836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:15.725857 24836 log.cc:824] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:15.728065 24836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28: No bootstrap required, opened a new log
Sep 26 12:41:15 dist-test-slave-m3pj krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:15.732532 24836 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9743e123299452da910132e9efccc28" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35119 } }
I20260926 12:41:15.732924 24836 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:15.733011 24836 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9743e123299452da910132e9efccc28, State: Initialized, Role: FOLLOWER
I20260926 12:41:15.733547 24836 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9743e123299452da910132e9efccc28" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35119 } }
I20260926 12:41:15.733692 24836 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:41:15.733781 24836 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:41:15.733893 24836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:15.734885 24836 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9743e123299452da910132e9efccc28" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35119 } }
I20260926 12:41:15.735275 24836 leader_election.cc:302] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9743e123299452da910132e9efccc28; no voters:
I20260926 12:41:15.735710 24836 leader_election.cc:288] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:41:15.735817 24841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:15.736040 24841 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [term 1 LEADER]: Becoming Leader. State: Replica: f9743e123299452da910132e9efccc28, State: Running, Role: LEADER
I20260926 12:41:15.736403 24841 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9743e123299452da910132e9efccc28" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35119 } }
I20260926 12:41:15.737195 24836 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:15.738274 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:15.722903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.10.224.190:35119)
0926 12:41:15.723447 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:15.723464 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:15.723505 (+ 41us) server_negotiation.cc:407] Connection header received
0926 12:41:15.724301 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:15.724321 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:15.724691 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:15.725065 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:15.725962 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:15.727021 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:15.728054 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:15.728646 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:15.728825 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:15.731965 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:15.731972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:15.731989 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:15.732023 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:15.734493 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:15.735036 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:15.735043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:15.735051 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:15.735198 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:15.735556 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:15.735561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:15.735563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:15.736225 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:15.736451 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:15.736758 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 12:41:15.737094 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":160,"threads_started":1}
I20260926 12:41:15.738248 24842 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9743e123299452da910132e9efccc28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9743e123299452da910132e9efccc28" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35119 } } }
I20260926 12:41:15.738380 24842 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:15.738947 24843 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9743e123299452da910132e9efccc28. Latest consensus state: current_term: 1 leader_uuid: "f9743e123299452da910132e9efccc28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9743e123299452da910132e9efccc28" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 35119 } } }
I20260926 12:41:15.739060 24843 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:15.739346 24850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:15.742168 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:41:15.742873 24850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:15.749331 24850 catalog_manager.cc:1347] Generated new cluster ID: 959a4d750265454aa0b6752b3780c067
I20260926 12:41:15.749450 24850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:41:15.760831 24850 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:41:15.762126 24850 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:41:15.771066 24850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9743e123299452da910132e9efccc28: Generated new TSK 0
I20260926 12:41:15.772049 24850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:41:15.798785 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 24768
2026-09-26T12:41:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1466 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:41:15 dist-test-slave-m3pj krb5kdc[24863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:41:15 dist-test-slave-m3pj krb5kdc[24863](info): set up 2 sockets
Sep 26 12:41:15 dist-test-slave-m3pj krb5kdc[24863](info): commencing operation
krb5kdc: starting...
W20260926 12:41:16.848215 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:41:16 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:41:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:41:16.979118 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38441
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:17.097393 24876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:17.097702 24876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:17.097744 24876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:17.102455 24876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:17.102531 24876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:17.102551 24876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:17.102568 24876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:17.102584 24876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:17.108284 24876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:38441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:17.109380 24876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:17.110395 24876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:17.119236 24884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:17.119236 24881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:17.119251 24882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:17.119326 24876 server_base.cc:1034] running on GCE node
I20260926 12:41:17.120218 24876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:17.121347 24876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:17.122562 24876 hybrid_clock.cc:648] HybridClock initialized: now 1790426477122519 us; error 57 us; skew 500 ppm
Sep 26 12:41:17 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:17.125921 24876 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:17.126974 24876 webserver.cc:460] Webserver started at http://127.10.224.190:40947/ using document root <none> and password file <none>
I20260926 12:41:17.127655 24876 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:17.127755 24876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:17.128008 24876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:17.130306 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "69251a29accb4744b298e389fa937fd8"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "605e154befe9e2a9e77be92d90686b01"
server_key_iv: "9969a93a99da79640df46e44e5370cf8"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.130934 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "69251a29accb4744b298e389fa937fd8"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "605e154befe9e2a9e77be92d90686b01"
server_key_iv: "9969a93a99da79640df46e44e5370cf8"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.135097 24876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:41:17.138415 24891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.140161 24876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:17.140345 24876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "69251a29accb4744b298e389fa937fd8"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "605e154befe9e2a9e77be92d90686b01"
server_key_iv: "9969a93a99da79640df46e44e5370cf8"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.140511 24876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:17.158003 24876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:17.161835 24876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:17.162107 24876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:17.169926 24876 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:38441
I20260926 12:41:17.170282 24943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:38441 every 8 connection(s)
I20260926 12:41:17.171030 24876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:17.174046 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:17.176781 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 24876
I20260926 12:41:17.176934 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:41:17.177250 11138 external_mini_cluster.cc:1428] Setting key 4a743f61c5c3c883cd51c307ba42412b
I20260926 12:41:17.181356 24944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Bootstrap starting.
I20260926 12:41:17.184348 24944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:17.185379 24944 log.cc:824] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:41:17 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:17.187969 24944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: No bootstrap required, opened a new log
I20260926 12:41:17.192268 24944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:17.192553 24944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:17.192642 24944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69251a29accb4744b298e389fa937fd8, State: Initialized, Role: FOLLOWER
I20260926 12:41:17.193104 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:17.178750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.10.224.190:38441)
0926 12:41:17.179208 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:17.179225 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:17.179263 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:41:17.180007 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:17.180027 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:17.180382 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:17.180686 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:17.181752 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.182798 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.184146 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.184723 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.184848 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:17.187891 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:17.187897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:17.187911 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:17.187940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:17.189897 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:17.190395 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:17.190406 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:17.190411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:17.190530 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:17.190825 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:17.190830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:17.190831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:17.191331 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:17.191592 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:17.191895 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 12:41:17.192212 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":156,"threads_started":1}
I20260926 12:41:17.193265 24944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:17.193456 24944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:41:17.193557 24944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:41:17.193672 24944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:17.195003 24944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:17.195426 24944 leader_election.cc:302] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69251a29accb4744b298e389fa937fd8; no voters:
I20260926 12:41:17.195873 24944 leader_election.cc:288] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:41:17.195961 24949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:17.196175 24949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 1 LEADER]: Becoming Leader. State: Replica: 69251a29accb4744b298e389fa937fd8, State: Running, Role: LEADER
I20260926 12:41:17.196507 24949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:17.197329 24944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:17.198504 24950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69251a29accb4744b298e389fa937fd8. Latest consensus state: current_term: 1 leader_uuid: "69251a29accb4744b298e389fa937fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } } }
I20260926 12:41:17.198513 24951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69251a29accb4744b298e389fa937fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } } }
I20260926 12:41:17.198679 24951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:17.198685 24950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:17.199095 24958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:17.202817 24958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:17.209836 24958 catalog_manager.cc:1347] Generated new cluster ID: 327f86d5c51f45a0b0bf9b06e079be19
I20260926 12:41:17.209939 24958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:41:17.217159 24958 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:41:17.218509 24958 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:41:17.224917 24958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Generated new TSK 0
I20260926 12:41:17.225831 24958 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:41:17.275836 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38441
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:17.415369 24970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:17.415748 24970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:17.415835 24970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:17.420076 24970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:17.420166 24970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:17.420257 24970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:41:17.425768 24970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:17.426903 24970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:17.427996 24970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:17.438722 24976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:17.438804 24970 server_base.cc:1034] running on GCE node
W20260926 12:41:17.438714 24975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:17.438964 24978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:17.439380 24970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:17.440153 24970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:17.441354 24970 hybrid_clock.cc:648] HybridClock initialized: now 1790426477441338 us; error 34 us; skew 500 ppm
Sep 26 12:41:17 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:17.444408 24970 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:41:17.445379 24970 webserver.cc:460] Webserver started at http://127.10.224.129:46165/ using document root <none> and password file <none>
I20260926 12:41:17.445992 24970 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:17.446089 24970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:17.446327 24970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:17.448490 24970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "cc27ac88a593467585e479e8e763cbaf"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "633288bef6e16d00844162a90d3d9100"
server_key_iv: "ad59d64b845ccbfff10decd51936d247"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.449075 24970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "cc27ac88a593467585e479e8e763cbaf"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "633288bef6e16d00844162a90d3d9100"
server_key_iv: "ad59d64b845ccbfff10decd51936d247"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.453016 24970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:41:17.455564 24985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.456923 24970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:41:17.457062 24970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "cc27ac88a593467585e479e8e763cbaf"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "633288bef6e16d00844162a90d3d9100"
server_key_iv: "ad59d64b845ccbfff10decd51936d247"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.457183 24970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:17.487890 24970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:17.491207 24970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:17.491441 24970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:17.492115 24970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:17.492995 24970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:17.493072 24970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.493158 24970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:17.493192 24970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.503378 24970 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:42237
I20260926 12:41:17.503396 25098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:42237 every 8 connection(s)
I20260926 12:41:17.504388 24970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:41:17.514253 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 24970
I20260926 12:41:17.514407 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
Sep 26 12:41:17 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:17.514750 11138 external_mini_cluster.cc:1428] Setting key 4918a294dccb472aae6b48832717bb2a
I20260926 12:41:17.519719 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:17.506244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:55217 (local address 127.10.224.190:38441)
0926 12:41:17.506409 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:17.506414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:17.507419 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 12:41:17.508927 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:17.508930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:17.508979 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:17.509052 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:17.510225 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.510767 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.512049 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.512368 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.512464 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:17.515195 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:17.515198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:17.515200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:17.515226 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:17.516934 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:17.517617 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:17.517620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:17.517621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:17.517704 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:17.518188 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:17.518193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:17.518196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:17.518523 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:17.518639 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:17.519280 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 12:41:17.519436 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:41:17.520637 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:17.506687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:55217)
0926 12:41:17.507250 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:17.507296 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:17.508715 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:17.509226 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:17.509233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:17.509599 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:17.510078 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:17.510091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.510917 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.510920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:17.511908 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:17.511917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.512524 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.512527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:17.512674 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:17.513348 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:17.513364 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:17.515011 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:17.517064 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:17.517073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:17.517094 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:17.517489 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:17.517817 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:17.517823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:17.517828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:17.518064 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:17.518652 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:17.518658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:17.518945 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 12:41:17.519201 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 12:41:17.519499 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260926 12:41:17.522022 25099 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38441
I20260926 12:41:17.522300 25099 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:17.522827 25099 heartbeater.cc:511] Master 127.10.224.190:38441 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:41:17.524457 24908 ts_manager.cc:194] Registered new tserver with Master: cc27ac88a593467585e479e8e763cbaf (127.10.224.129:42237)
I20260926 12:41:17.525843 24908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:55217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:41:17.543984 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38441
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:17.657591 25104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:17.657928 25104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:17.658017 25104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:17.662461 25104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:17.662555 25104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:17.662648 25104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:41:17.668239 25104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:17.669440 25104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:17.670480 25104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:17.680364 25110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:17.680323 25112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:17.680342 25109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:17.680732 25104 server_base.cc:1034] running on GCE node
I20260926 12:41:17.681152 25104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:17.681774 25104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:17.682981 25104 hybrid_clock.cc:648] HybridClock initialized: now 1790426477682964 us; error 28 us; skew 500 ppm
Sep 26 12:41:17 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:17.685786 25104 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:41:17.686682 25104 webserver.cc:460] Webserver started at http://127.10.224.130:40455/ using document root <none> and password file <none>
I20260926 12:41:17.687271 25104 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:17.687362 25104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:17.687654 25104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:17.689713 25104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "b64ab430eba34be3bbda34f90f06d6f0"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "07a801309d491a3bf2632701f6f8d315"
server_key_iv: "f243bc71ab0db919c35c89bbecac8991"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.690330 25104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "b64ab430eba34be3bbda34f90f06d6f0"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "07a801309d491a3bf2632701f6f8d315"
server_key_iv: "f243bc71ab0db919c35c89bbecac8991"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.694135 25104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:41:17.696722 25119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.698191 25104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:41:17.698331 25104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "b64ab430eba34be3bbda34f90f06d6f0"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "07a801309d491a3bf2632701f6f8d315"
server_key_iv: "f243bc71ab0db919c35c89bbecac8991"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.698515 25104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:17.717530 25104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:17.721146 25104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:17.721374 25104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:17.722013 25104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:17.722927 25104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:17.722999 25104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.723066 25104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:17.723115 25104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.732757 25104 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:33779
I20260926 12:41:17.732764 25232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:33779 every 8 connection(s)
I20260926 12:41:17.733716 25104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:41:17.741004 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 25104
I20260926 12:41:17.741123 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:41:17.741464 11138 external_mini_cluster.cc:1428] Setting key 2d822b1ab7633011d8490d2bdcd2f93f
Sep 26 12:41:17 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:17.749070 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:17.735568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:53685 (local address 127.10.224.190:38441)
0926 12:41:17.735728 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:17.735732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:17.736639 (+ 907us) server_negotiation.cc:407] Connection header received
0926 12:41:17.738115 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:17.738118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:17.738170 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:17.738253 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:17.739448 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.740049 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.741303 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.741627 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.741730 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:17.744619 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:17.744623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:17.744626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:17.744659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:17.746404 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:17.747131 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:17.747134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:17.747135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:17.747209 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:17.747764 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:17.747767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:17.747768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:17.747958 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:17.748036 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:17.748827 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 12:41:17.748941 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:41:17.750556 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:17.736036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:53685)
0926 12:41:17.736484 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:17.736529 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:17.737912 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:17.738414 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:17.738421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:17.738815 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:17.739316 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:17.739329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.740162 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.740165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:17.741173 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:17.741190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.741792 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.741795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:17.741990 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:17.742647 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:17.742668 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:17.744438 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:17.746535 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:17.746546 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:17.746569 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:17.747021 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:17.747346 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:17.747352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:17.747356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:17.747655 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:17.748074 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:17.748085 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:17.748502 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 12:41:17.748783 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 12:41:17.749239 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260926 12:41:17.752067 25233 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38441
I20260926 12:41:17.752372 25233 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:41:17.753058 25233 heartbeater.cc:511] Master 127.10.224.190:38441 requested a full tablet report, sending...
I20260926 12:41:17.754354 24908 ts_manager.cc:194] Registered new tserver with Master: b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
I20260926 12:41:17.754987 24908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:53685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:41:17.774631 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38441
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:17.898277 25238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:17.898598 25238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:17.898674 25238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:17.903115 25238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:17.903187 25238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:17.903250 25238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:41:17.910737 25238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:17.911984 25238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:17.912989 25238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:17.923172 25246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:17.923273 25238 server_base.cc:1034] running on GCE node
W20260926 12:41:17.923166 25243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:17.923462 25244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:17.923880 25238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:17.924568 25238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:17.925762 25238 hybrid_clock.cc:648] HybridClock initialized: now 1790426477925746 us; error 29 us; skew 500 ppm
Sep 26 12:41:17 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:17.928678 25238 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:41:17.929651 25238 webserver.cc:460] Webserver started at http://127.10.224.131:36689/ using document root <none> and password file <none>
I20260926 12:41:17.930253 25238 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:17.930368 25238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:17.930606 25238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:17.932793 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "758872d19472487a85b61e8cbe991e61"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "858da58ab1166a04ffa2952e88023f15"
server_key_iv: "62e97e85ffdc88bd3825ff39351a28fa"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.933389 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "758872d19472487a85b61e8cbe991e61"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "858da58ab1166a04ffa2952e88023f15"
server_key_iv: "62e97e85ffdc88bd3825ff39351a28fa"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.937350 25238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:41:17.940162 25253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.941562 25238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:17.941707 25238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "758872d19472487a85b61e8cbe991e61"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "858da58ab1166a04ffa2952e88023f15"
server_key_iv: "62e97e85ffdc88bd3825ff39351a28fa"
server_key_version: "encryptionkey@0"
I20260926 12:41:17.941840 25238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:17.954260 25238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:17.958226 25238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:17.958467 25238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:17.959124 25238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:17.960127 25238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:17.960187 25238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.960297 25238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:17.960340 25238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:17.970249 25238 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:41239
I20260926 12:41:17.970289 25366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:41239 every 8 connection(s)
I20260926 12:41:17.971304 25238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:41:17.972126 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 25238
I20260926 12:41:17.972256 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:41:17.972565 11138 external_mini_cluster.cc:1428] Setting key afa78fa09b3c402ed588bf04a228153f
Sep 26 12:41:17 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:17.988165 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:17.974592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:36353 (local address 127.10.224.190:38441)
0926 12:41:17.974736 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:17.974739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:17.975891 (+ 1152us) server_negotiation.cc:407] Connection header received
0926 12:41:17.977364 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:17.977368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:17.977426 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:17.977506 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:17.978740 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.979294 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.980591 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.980827 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.980941 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:17.983966 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:17.983970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:17.983972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:17.983999 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:17.985516 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:17.986346 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:17.986349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:17.986350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:17.986464 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:17.986891 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:17.986895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:17.986896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:17.987148 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:17.987264 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:17.987929 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 12:41:17.988044 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:41:17.989147 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:17.975170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:36353)
0926 12:41:17.975735 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:17.975786 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:17.977146 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:17.977690 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:17.977698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:17.978094 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:17.978593 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:17.978608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.979435 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.979439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:17.980456 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:17.980466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:17.980990 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:17.980994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:17.981176 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:17.981968 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:17.981985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:17.983768 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:17.985715 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:17.985725 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:17.985744 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:17.986160 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:17.986575 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:17.986578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:17.986580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:17.986780 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:17.987301 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:17.987309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:17.987680 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 12:41:17.987874 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 12:41:17.988190 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260926 12:41:17.990304 25367 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38441
I20260926 12:41:17.990545 25367 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:17.991072 25367 heartbeater.cc:511] Master 127.10.224.190:38441 requested a full tablet report, sending...
I20260926 12:41:17.992154 24908 ts_manager.cc:194] Registered new tserver with Master: 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239)
I20260926 12:41:17.992765 24908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:36353
I20260926 12:41:17.999413 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:41:18.010677 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.001674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.10.224.190:38441)
0926 12:41:18.001819 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.001823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.002018 (+ 195us) server_negotiation.cc:407] Connection header received
0926 12:41:18.002117 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.002120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.002186 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.002281 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:18.003086 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.003742 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.004566 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.004803 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.004896 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.006501 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:18.006505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:18.006507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:18.006537 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:18.008571 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:18.009280 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:18.009284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:18.009285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:18.009398 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:18.009702 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:18.009705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:18.009706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:18.009971 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:18.010086 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.010415 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.010537 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:41:18.014513 24908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:41:18.017948 24908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:41:18.034396 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.027620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33779 (local address 127.0.0.1:48868)
0926 12:41:18.028361 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.028373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:18.028453 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:18.029988 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:18.029991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:18.030002 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:18.030091 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.030100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.031174 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.031180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.032632 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.032642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.033686 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.033690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.033819 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.033878 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 12:41:18.033996 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:41:18.034143 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":155,"thread_start_us":97,"threads_started":1}
I20260926 12:41:18.034412 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.028784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.10.224.129:42237)
0926 12:41:18.029106 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.029113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.029255 (+ 142us) server_negotiation.cc:407] Connection header received
0926 12:41:18.029487 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.029492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.029731 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.029916 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:18.030152 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.031326 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.033132 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.033818 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.033954 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.034094 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.034157 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.034265 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
I20260926 12:41:18.034759 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.028334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.10.224.130:33779)
0926 12:41:18.029545 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.029557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.029577 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:41:18.029650 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.029656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.029821 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.030230 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:18.030363 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.031040 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.032761 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.033542 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.033677 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.033823 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.034297 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.034394 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":139,"threads_started":1}
I20260926 12:41:18.034998 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.028311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42237 (local address 127.0.0.1:33632)
0926 12:41:18.029150 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.029163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:18.029298 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:18.029910 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:18.029916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:18.029938 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:18.030040 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.030046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.031446 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.031452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.032975 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.032986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.033949 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.033952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.034047 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.034071 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:41:18.034130 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 12:41:18.034210 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":71,"thread_start_us":99,"threads_started":1}
I20260926 12:41:18.035282 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.027483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:41239 (local address 127.0.0.1:34492)
0926 12:41:18.028165 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.028219 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:18.028353 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:18.029392 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:18.029400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:18.029421 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:18.029561 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.029570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.031207 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.031211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.032975 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.032986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.034231 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.034234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.034320 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.034339 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:41:18.034406 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:41:18.034481 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":102,"thread_start_us":134,"threads_started":1}
I20260926 12:41:18.035336 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.027628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.10.224.131:41239)
0926 12:41:18.028651 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.028666 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.028689 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:41:18.028777 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.028783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.029004 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.029161 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:18.029699 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.031025 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.033107 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.033936 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.034051 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.034174 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.035050 (+ 876us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.035180 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":167,"threads_started":1}
I20260926 12:41:18.038000 25033 tablet_service.cc:1467] Processing CreateTablet for tablet ff98fc3378d940fba58154404b70cf43 (DEFAULT_TABLE table=test-table [id=dc328e97bdcc49cca56d4a55a78007df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:18.038203 25301 tablet_service.cc:1467] Processing CreateTablet for tablet ff98fc3378d940fba58154404b70cf43 (DEFAULT_TABLE table=test-table [id=dc328e97bdcc49cca56d4a55a78007df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:18.038196 25167 tablet_service.cc:1467] Processing CreateTablet for tablet ff98fc3378d940fba58154404b70cf43 (DEFAULT_TABLE table=test-table [id=dc328e97bdcc49cca56d4a55a78007df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:18.039220 25033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff98fc3378d940fba58154404b70cf43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:18.039314 25167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff98fc3378d940fba58154404b70cf43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:18.039319 25301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff98fc3378d940fba58154404b70cf43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:18.046192 25384 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Bootstrap starting.
I20260926 12:41:18.048044 25386 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Bootstrap starting.
I20260926 12:41:18.048730 25384 tablet_bootstrap.cc:654] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:18.049486 25384 log.cc:824] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:18.050213 25385 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Bootstrap starting.
I20260926 12:41:18.050712 25386 tablet_bootstrap.cc:654] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:18.051651 25386 log.cc:824] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:18.052276 25384 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: No bootstrap required, opened a new log
I20260926 12:41:18.052520 25384 ts_tablet_manager.cc:1397] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:41:18.052762 25385 tablet_bootstrap.cc:654] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:18.053464 25385 log.cc:824] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:18.054006 25386 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: No bootstrap required, opened a new log
I20260926 12:41:18.054217 25386 ts_tablet_manager.cc:1397] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:41:18.055229 25385 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: No bootstrap required, opened a new log
I20260926 12:41:18.055435 25385 ts_tablet_manager.cc:1397] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:41:18.057775 25384 raft_consensus.cc:348] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.058099 25384 raft_consensus.cc:374] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:18.058164 25384 raft_consensus.cc:729] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc27ac88a593467585e479e8e763cbaf, State: Initialized, Role: FOLLOWER
I20260926 12:41:18.058691 25384 consensus_queue.cc:260] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.058583 25386 raft_consensus.cc:348] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.058846 25386 raft_consensus.cc:374] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:18.058702 25385 raft_consensus.cc:348] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.058908 25386 raft_consensus.cc:729] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64ab430eba34be3bbda34f90f06d6f0, State: Initialized, Role: FOLLOWER
I20260926 12:41:18.058943 25385 raft_consensus.cc:374] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:18.058995 25385 raft_consensus.cc:729] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758872d19472487a85b61e8cbe991e61, State: Initialized, Role: FOLLOWER
I20260926 12:41:18.059458 25099 heartbeater.cc:503] Master 127.10.224.190:38441 was elected leader, sending a full tablet report...
I20260926 12:41:18.059406 25386 consensus_queue.cc:260] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.059934 25384 ts_tablet_manager.cc:1428] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 12:41:18.060451 25386 ts_tablet_manager.cc:1428] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:41:18.060609 25233 heartbeater.cc:503] Master 127.10.224.190:38441 was elected leader, sending a full tablet report...
I20260926 12:41:18.061617 25385 consensus_queue.cc:260] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.062407 25367 heartbeater.cc:503] Master 127.10.224.190:38441 was elected leader, sending a full tablet report...
I20260926 12:41:18.062667 25385 ts_tablet_manager.cc:1428] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:41:18.190577 25390 raft_consensus.cc:482] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:18.190883 25390 raft_consensus.cc:504] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.192236 25390 leader_election.cc:288] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239), b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
I20260926 12:41:18.196735 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.192603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:50697 (local address 127.10.224.131:41239)
0926 12:41:18.192776 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.192783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.193051 (+ 268us) server_negotiation.cc:407] Connection header received
0926 12:41:18.193160 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.193164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.193224 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.193314 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:18.193656 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.194211 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.195435 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.195903 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.196154 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.196287 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.196482 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.196574 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:41:18.196729 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.192477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:41239 (local address 127.10.224.129:50697)
0926 12:41:18.192939 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.192952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:18.193063 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:18.193338 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:18.193341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:18.193356 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:18.193525 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.193533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.194400 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.194403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.195302 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.195312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.196161 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.196166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.196290 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.196318 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:41:18.196437 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 12:41:18.196538 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":25,"thread_start_us":133,"threads_started":1}
I20260926 12:41:18.196907 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.192688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:52101 (local address 127.10.224.130:33779)
0926 12:41:18.192861 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.192866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.193168 (+ 302us) server_negotiation.cc:407] Connection header received
0926 12:41:18.193317 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.193320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.193377 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.193464 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:18.193751 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.194252 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.195437 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.196196 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.196361 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.196434 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.196681 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.196773 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 12:41:18.196959 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.192601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33779 (local address 127.10.224.129:52101)
0926 12:41:18.193046 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.193060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:18.193174 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:18.193486 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:18.193488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:18.193503 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:18.193625 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.193634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.194381 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.194386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.195300 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.195311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.196383 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.196390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.196534 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.196562 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:41:18.196645 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:41:18.196716 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":23,"thread_start_us":151,"threads_started":1}
I20260926 12:41:18.197860 25321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff98fc3378d940fba58154404b70cf43" candidate_uuid: "cc27ac88a593467585e479e8e763cbaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758872d19472487a85b61e8cbe991e61" is_pre_election: true
I20260926 12:41:18.197863 25187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff98fc3378d940fba58154404b70cf43" candidate_uuid: "cc27ac88a593467585e479e8e763cbaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64ab430eba34be3bbda34f90f06d6f0" is_pre_election: true
I20260926 12:41:18.198246 25321 raft_consensus.cc:2454] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc27ac88a593467585e479e8e763cbaf in term 0.
I20260926 12:41:18.198298 25187 raft_consensus.cc:2454] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc27ac88a593467585e479e8e763cbaf in term 0.
I20260926 12:41:18.198874 24986 leader_election.cc:302] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758872d19472487a85b61e8cbe991e61, cc27ac88a593467585e479e8e763cbaf; no voters:
I20260926 12:41:18.199143 25390 raft_consensus.cc:2781] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:41:18.199262 25390 raft_consensus.cc:482] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:41:18.199388 25390 raft_consensus.cc:3037] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:18.200742 25390 raft_consensus.cc:504] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.201182 25390 leader_election.cc:288] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 1 election: Requested vote from peers 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239), b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
I20260926 12:41:18.201727 25187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff98fc3378d940fba58154404b70cf43" candidate_uuid: "cc27ac88a593467585e479e8e763cbaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64ab430eba34be3bbda34f90f06d6f0"
I20260926 12:41:18.201727 25321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff98fc3378d940fba58154404b70cf43" candidate_uuid: "cc27ac88a593467585e479e8e763cbaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758872d19472487a85b61e8cbe991e61"
I20260926 12:41:18.201874 25321 raft_consensus.cc:3037] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:18.201903 25187 raft_consensus.cc:3037] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:18.203326 25187 raft_consensus.cc:2454] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc27ac88a593467585e479e8e763cbaf in term 1.
I20260926 12:41:18.203326 25321 raft_consensus.cc:2454] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc27ac88a593467585e479e8e763cbaf in term 1.
I20260926 12:41:18.203717 24989 leader_election.cc:302] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b64ab430eba34be3bbda34f90f06d6f0, cc27ac88a593467585e479e8e763cbaf; no voters:
I20260926 12:41:18.203940 25390 raft_consensus.cc:2781] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:18.204229 25390 raft_consensus.cc:686] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 1 LEADER]: Becoming Leader. State: Replica: cc27ac88a593467585e479e8e763cbaf, State: Running, Role: LEADER
I20260926 12:41:18.204627 25390 consensus_queue.cc:237] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.208112 24908 catalog_manager.cc:5495] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf reported cstate change: term changed from 0 to 1, leader changed from <none> to cc27ac88a593467585e479e8e763cbaf (127.10.224.129). New cstate: current_term: 1 leader_uuid: "cc27ac88a593467585e479e8e763cbaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } health_report { overall_health: HEALTHY } } }
W20260926 12:41:18.223580 25368 tablet.cc:2369] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:41:18.225739 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.222256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.10.224.129:42237)
0926 12:41:18.222401 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.222404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.222437 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:41:18.222514 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.222516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.222571 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.222650 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:41:18.222911 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.223492 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.224407 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.224621 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.224710 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.224976 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:41:18.225407 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:41:18.225499 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.225560 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.225610 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:41:18.231964 25321 raft_consensus.cc:1261] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 1 FOLLOWER]: Refusing update from remote peer cc27ac88a593467585e479e8e763cbaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:18.231964 25187 raft_consensus.cc:1261] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 1 FOLLOWER]: Refusing update from remote peer cc27ac88a593467585e479e8e763cbaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:18.232800 25395 consensus_queue.cc:1035] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:18.233040 25390 consensus_queue.cc:1035] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:41:18.234886 25234 tablet.cc:2369] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:41:18.242326 25399 mvcc.cc:205] Tried to move back new op lower bound from 7333586854829883392 to 7333586854729461760. Current Snapshot: MvccSnapshot[applied={T|T < 7333586854829883392}]
I20260926 12:41:18.252620 24906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:18.252856 24906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:18.257254 24906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Sending DeleteTablet for 3 replicas of tablet ff98fc3378d940fba58154404b70cf43
I20260926 12:41:18.261368 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 24970
I20260926 12:41:18.261381 25167 tablet_service.cc:1514] Processing DeleteTablet for tablet ff98fc3378d940fba58154404b70cf43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:18 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:48868
I20260926 12:41:18.262878 25301 tablet_service.cc:1514] Processing DeleteTablet for tablet ff98fc3378d940fba58154404b70cf43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:18 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:34492
I20260926 12:41:18.263597 25409 tablet_replica.cc:330] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: stopping tablet replica
I20260926 12:41:18.263929 25408 tablet_replica.cc:330] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: stopping tablet replica
I20260926 12:41:18.264014 25409 raft_consensus.cc:2229] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:18.264429 25409 raft_consensus.cc:2258] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:18.264415 25408 raft_consensus.cc:2229] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:18.264844 25408 raft_consensus.cc:2258] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:18.266464 25409 ts_tablet_manager.cc:1905] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:18.266700 25408 ts_tablet_manager.cc:1905] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:18.269555 25408 ts_tablet_manager.cc:1918] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:18.269758 25408 log.cc:1196] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/ff98fc3378d940fba58154404b70cf43
I20260926 12:41:18.269941 25409 ts_tablet_manager.cc:1918] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:18.270042 25409 log.cc:1196] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/ff98fc3378d940fba58154404b70cf43
I20260926 12:41:18.270226 25408 ts_tablet_manager.cc:1939] T ff98fc3378d940fba58154404b70cf43 P b64ab430eba34be3bbda34f90f06d6f0: Deleting consensus metadata
I20260926 12:41:18.270393 25409 ts_tablet_manager.cc:1939] T ff98fc3378d940fba58154404b70cf43 P 758872d19472487a85b61e8cbe991e61: Deleting consensus metadata
I20260926 12:41:18.271997 24892 catalog_manager.cc:4841] TS 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239): tablet ff98fc3378d940fba58154404b70cf43 (table test-table [id=dc328e97bdcc49cca56d4a55a78007df]) successfully deleted
I20260926 12:41:18.272027 24895 catalog_manager.cc:4841] TS b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779): tablet ff98fc3378d940fba58154404b70cf43 (table test-table [id=dc328e97bdcc49cca56d4a55a78007df]) successfully deleted
W20260926 12:41:18.272715 24893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:42237 (error 108)
I20260926 12:41:18.273013 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 25104
I20260926 12:41:18.275190 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.274854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42237 (local address 127.0.0.1:33636)
0926 12:41:18.274998 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.275080 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:42237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 12:41:18.275384 24893 catalog_manager.cc:4584] TS cc27ac88a593467585e479e8e763cbaf (127.10.224.129:42237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff98fc3378d940fba58154404b70cf43: Network error: Client connection negotiation failed: client connection to 127.10.224.129:42237: connect: Connection refused (error 111)
I20260926 12:41:18.284341 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 25238
I20260926 12:41:18.291466 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 24876
I20260926 12:41:18.298395 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38441
--webserver_interface=127.10.224.190
--webserver_port=40947
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:18.416101 25411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:18.416460 25411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:18.416535 25411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:18.420792 25411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:18.420904 25411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:18.420943 25411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:18.420979 25411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:18.421015 25411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:18.426697 25411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:38441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:18.427959 25411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:18.429013 25411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:18.443599 25420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:18.443683 25411 server_base.cc:1034] running on GCE node
W20260926 12:41:18.443610 25418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:18.443609 25417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:18.444509 25411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:18.445868 25411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:18.447050 25411 hybrid_clock.cc:648] HybridClock initialized: now 1790426478447030 us; error 31 us; skew 500 ppm
Sep 26 12:41:18 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:18.450112 25411 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:18.451238 25411 webserver.cc:460] Webserver started at http://127.10.224.190:40947/ using document root <none> and password file <none>
I20260926 12:41:18.451968 25411 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:18.452095 25411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:18.456430 25411 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:41:18.458832 25427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:18.460443 25411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:18.460623 25411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "69251a29accb4744b298e389fa937fd8"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "605e154befe9e2a9e77be92d90686b01"
server_key_iv: "9969a93a99da79640df46e44e5370cf8"
server_key_version: "encryptionkey@0"
I20260926 12:41:18.461321 25411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:18.495787 25411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:18.499351 25411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:18.499701 25411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:18.507648 25411 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:38441
I20260926 12:41:18.507978 25479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:38441 every 8 connection(s)
I20260926 12:41:18.508934 25411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:18.513638 25480 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:41:18.515393 25480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Bootstrap starting.
I20260926 12:41:18.516209 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 25411
I20260926 12:41:18.516706 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:42237
--local_ip_for_outbound_sockets=127.10.224.129
--tserver_master_addrs=127.10.224.190:38441
--webserver_port=46165
--webserver_interface=127.10.224.129
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:18.534075 25480 log.cc:824] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:18.543121 25480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:41:18.543599 25480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Bootstrap complete.
I20260926 12:41:18.547827 25480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:18.548257 25480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69251a29accb4744b298e389fa937fd8, State: Initialized, Role: FOLLOWER
I20260926 12:41:18.548767 25480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:18.548892 25480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:41:18.548985 25480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:41:18.549101 25480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:41:18.551077 25480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:18.551484 25480 leader_election.cc:302] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69251a29accb4744b298e389fa937fd8; no voters:
I20260926 12:41:18.551890 25480 leader_election.cc:288] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:41:18.551995 25484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:41:18.552203 25484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 2 LEADER]: Becoming Leader. State: Replica: 69251a29accb4744b298e389fa937fd8, State: Running, Role: LEADER
I20260926 12:41:18.552517 25484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:18.553181 25480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:18.553514 25484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69251a29accb4744b298e389fa937fd8. Latest consensus state: current_term: 2 leader_uuid: "69251a29accb4744b298e389fa937fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } } }
I20260926 12:41:18.553633 25484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:18.553995 25488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:18.554085 25485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "69251a29accb4744b298e389fa937fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } } }
I20260926 12:41:18.554271 25485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:18.559054 25488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:18.559741 25488 catalog_manager.cc:1259] Loaded cluster ID: 327f86d5c51f45a0b0bf9b06e079be19
I20260926 12:41:18.559808 25488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:41:18.561216 25488 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 12:41:18.566130 25488 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 12:41:18.572306 25488 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:41:18.573176 25488 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Loaded TSK: 0
I20260926 12:41:18.574432 25488 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:41:18.638832 25482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:18.639276 25482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:18.639359 25482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:18.644126 25482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:18.644228 25482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:18.644330 25482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:41:18.650632 25482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:42237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=46165
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:18.651916 25482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:18.653164 25482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:18.664659 25507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:18.664673 25508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:18.664849 25482 server_base.cc:1034] running on GCE node
W20260926 12:41:18.665144 25510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:18.665589 25482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:18.666370 25482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:18.667600 25482 hybrid_clock.cc:648] HybridClock initialized: now 1790426478667569 us; error 50 us; skew 500 ppm
Sep 26 12:41:18 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:18.670506 25482 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:41:18.671620 25482 webserver.cc:460] Webserver started at http://127.10.224.129:46165/ using document root <none> and password file <none>
I20260926 12:41:18.672221 25482 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:18.672300 25482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:18.676574 25482 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:41:18.678916 25517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:18.680338 25482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:41:18.680474 25482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "cc27ac88a593467585e479e8e763cbaf"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "633288bef6e16d00844162a90d3d9100"
server_key_iv: "ad59d64b845ccbfff10decd51936d247"
server_key_version: "encryptionkey@0"
I20260926 12:41:18.681051 25482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:18.710626 25482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:18.714152 25482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:18.714408 25482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:18.715080 25482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:18.716233 25524 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:41:18.720551 25482 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:41:18.720611 25482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 12:41:18.720687 25482 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:41:18.722414 25482 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:41:18.722461 25482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:41:18.722779 25524 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Bootstrap starting.
I20260926 12:41:18.733317 25482 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:42237
I20260926 12:41:18.733355 25631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:42237 every 8 connection(s)
I20260926 12:41:18.734411 25482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:41:18.734876 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 25482
I20260926 12:41:18.735412 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:33779
--local_ip_for_outbound_sockets=127.10.224.130
--tserver_master_addrs=127.10.224.190:38441
--webserver_port=40455
--webserver_interface=127.10.224.130
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:18.744472 25524 log.cc:824] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:41:18 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:18.755007 25524 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:41:18.755532 25524 tablet_bootstrap.cc:492] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Bootstrap complete.
I20260926 12:41:18.755766 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.736700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:53259)
0926 12:41:18.737461 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.737517 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:18.738335 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:18.739586 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:18.739629 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:18.740118 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:18.740738 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.740770 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.742362 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.742369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.743978 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.743990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.745097 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.745104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.745342 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.746146 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:18.746175 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:18.748338 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:18.751986 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:18.751993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:18.752007 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:18.752361 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:18.752819 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:18.752822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:18.752824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:18.753001 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:18.753877 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:18.753883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:18.754164 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 12:41:18.754348 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 12:41:18.754735 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":131,"threads_started":1}
I20260926 12:41:18.756589 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.736656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:53259 (local address 127.10.224.190:38441)
0926 12:41:18.737180 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.737199 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.738581 (+ 1382us) server_negotiation.cc:407] Connection header received
0926 12:41:18.738814 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.738823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.739130 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.739398 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:18.740942 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.742155 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.744130 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.744680 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.744895 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.748599 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:18.748610 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:18.748629 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:18.748668 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:18.751049 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:18.752529 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:18.752536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:18.752542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:18.752705 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:18.753122 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:18.753126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:18.753128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:18.753676 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:18.753809 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.754525 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.754851 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":112,"threads_started":1}
I20260926 12:41:18.756969 25524 ts_tablet_manager.cc:1397] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.008s
I20260926 12:41:18.759095 25632 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38441
I20260926 12:41:18.759351 25632 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:18.759979 25632 heartbeater.cc:511] Master 127.10.224.190:38441 requested a full tablet report, sending...
I20260926 12:41:18.759838 25524 raft_consensus.cc:348] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.760381 25524 raft_consensus.cc:729] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc27ac88a593467585e479e8e763cbaf, State: Initialized, Role: FOLLOWER
I20260926 12:41:18.760881 25524 consensus_queue.cc:260] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:18.761993 25524 ts_tablet_manager.cc:1428] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:41:18.764112 25444 ts_manager.cc:194] Registered new tserver with Master: cc27ac88a593467585e479e8e763cbaf (127.10.224.129:42237)
I20260926 12:41:18.768585 25444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:53259
I20260926 12:41:18.771489 25632 heartbeater.cc:503] Master 127.10.224.190:38441 was elected leader, sending a full tablet report...
Sep 26 12:41:18 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:41:18.779031 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.767593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.10.224.129:42237)
0926 12:41:18.768324 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.768334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.768353 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:41:18.768435 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.768439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.768592 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.768809 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:18.769506 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.770561 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.771965 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.772450 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.772554 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.775314 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:18.775323 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:18.775333 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:18.775356 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:18.777108 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:18.777653 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:18.777657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:18.777659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:18.777743 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:18.778021 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:18.778023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:18.778024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:18.778275 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:18.778380 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.778758 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.778901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":634,"thread_start_us":140,"threads_started":1}
I20260926 12:41:18.779071 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.767498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42237 (local address 127.0.0.1:33638)
0926 12:41:18.767939 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.767982 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:18.768111 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:18.768834 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:18.768839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:18.769203 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:18.769307 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.769340 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.770700 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.770704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.771834 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.771848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.772587 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.772591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.772701 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.773519 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:18.773534 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:18.775152 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:18.777220 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:18.777231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:18.777237 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:18.777524 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:18.777834 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:18.777837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:18.777839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:18.777934 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:18.778385 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:18.778391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:18.778530 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 12:41:18.778675 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 12:41:18.778892 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":131,"threads_started":1}
I20260926 12:41:18.780120 25566 tablet_service.cc:1514] Processing DeleteTablet for tablet ff98fc3378d940fba58154404b70cf43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:18 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:33638
I20260926 12:41:18.781015 25641 tablet_replica.cc:330] stopping tablet replica
I20260926 12:41:18.781344 25641 raft_consensus.cc:2229] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:18.781601 25641 raft_consensus.cc:2258] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:18.782846 25641 ts_tablet_manager.cc:1905] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:18.787595 25641 ts_tablet_manager.cc:1918] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:18.787704 25641 log.cc:1196] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/ff98fc3378d940fba58154404b70cf43
I20260926 12:41:18.788012 25641 ts_tablet_manager.cc:1939] T ff98fc3378d940fba58154404b70cf43 P cc27ac88a593467585e479e8e763cbaf: Deleting consensus metadata
I20260926 12:41:18.790436 25429 catalog_manager.cc:4841] TS cc27ac88a593467585e479e8e763cbaf (127.10.224.129:42237): tablet ff98fc3378d940fba58154404b70cf43 (table test-table [id=dc328e97bdcc49cca56d4a55a78007df]) successfully deleted
W20260926 12:41:18.866714 25635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:18.867024 25635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:18.867071 25635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:18.871236 25635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:18.871305 25635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:18.871371 25635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:41:18.876801 25635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:33779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=40455
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:18.878048 25635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:18.879070 25635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:18.889317 25647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:18.889277 25649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:18.889513 25635 server_base.cc:1034] running on GCE node
W20260926 12:41:18.889278 25646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:18.890110 25635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:18.890830 25635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:18.892024 25635 hybrid_clock.cc:648] HybridClock initialized: now 1790426478892008 us; error 28 us; skew 500 ppm
Sep 26 12:41:18 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:18.894914 25635 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:41:18.895972 25635 webserver.cc:460] Webserver started at http://127.10.224.130:40455/ using document root <none> and password file <none>
I20260926 12:41:18.896593 25635 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:18.896714 25635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:18.900786 25635 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 12:41:18.903020 25656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:18.904371 25635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:41:18.904510 25635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "b64ab430eba34be3bbda34f90f06d6f0"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "07a801309d491a3bf2632701f6f8d315"
server_key_iv: "f243bc71ab0db919c35c89bbecac8991"
server_key_version: "encryptionkey@0"
I20260926 12:41:18.905082 25635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:18.919798 25635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:18.922937 25635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:18.923166 25635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:18.923882 25635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:18.924752 25635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:18.924823 25635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:18.924899 25635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:18.924952 25635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:18.934324 25635 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:33779
I20260926 12:41:18.934341 25769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:33779 every 8 connection(s)
I20260926 12:41:18.935423 25635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:41:18.944700 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 25635
I20260926 12:41:18.945165 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:41239
--local_ip_for_outbound_sockets=127.10.224.131
--tserver_master_addrs=127.10.224.190:38441
--webserver_port=36689
--webserver_interface=127.10.224.131
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:41:18 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:18.950120 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.937338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:51593 (local address 127.10.224.190:38441)
0926 12:41:18.937513 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:18.937519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:18.938410 (+ 891us) server_negotiation.cc:407] Connection header received
0926 12:41:18.939818 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:18.939821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:18.939883 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:18.939961 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:18.941297 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.941839 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.943009 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.943235 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.943312 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.946017 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:18.946024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:18.946030 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:18.946066 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:18.947701 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:18.948335 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:18.948337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:18.948339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:18.948415 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:18.948865 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:18.948867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:18.948868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:18.949068 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:18.949149 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:18.949872 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 12:41:18.949995 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:41:18.951484 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:18.937762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:51593)
0926 12:41:18.938213 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:18.938297 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:18.939602 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:18.940262 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:18.940270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:18.940654 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:18.941142 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.941155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.941948 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.941950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.942899 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:18.942909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:18.943373 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:18.943376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:18.943551 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:18.944204 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:18.944227 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:18.945823 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:18.947817 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:18.947826 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:18.947848 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:18.948221 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:18.948507 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:18.948513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:18.948518 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:18.948716 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:18.949183 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:18.949193 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:18.949574 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 12:41:18.949831 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 12:41:18.950247 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260926 12:41:18.952785 25770 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38441
I20260926 12:41:18.953092 25770 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:18.953606 25770 heartbeater.cc:511] Master 127.10.224.190:38441 requested a full tablet report, sending...
I20260926 12:41:18.954800 25444 ts_manager.cc:194] Registered new tserver with Master: b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
I20260926 12:41:18.955750 25444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:51593
W20260926 12:41:19.062354 25773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:19.062644 25773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:19.062690 25773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:19.067065 25773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:19.067142 25773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:19.067205 25773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:41:19.073132 25773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:41239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=36689
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:19.074200 25773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:19.075212 25773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:19.085626 25781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:19.085626 25778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:19.085736 25773 server_base.cc:1034] running on GCE node
W20260926 12:41:19.085641 25779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:19.086376 25773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:19.087100 25773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:19.088318 25773 hybrid_clock.cc:648] HybridClock initialized: now 1790426479088280 us; error 53 us; skew 500 ppm
Sep 26 12:41:19 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426479, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:19.091573 25773 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:41:19.092607 25773 webserver.cc:460] Webserver started at http://127.10.224.131:36689/ using document root <none> and password file <none>
I20260926 12:41:19.093231 25773 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:19.093350 25773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:19.097643 25773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:41:19.100061 25788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:19.101620 25773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:19.101784 25773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "758872d19472487a85b61e8cbe991e61"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "858da58ab1166a04ffa2952e88023f15"
server_key_iv: "62e97e85ffdc88bd3825ff39351a28fa"
server_key_version: "encryptionkey@0"
I20260926 12:41:19.102474 25773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:19.130977 25773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:19.134316 25773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:19.134538 25773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:19.135177 25773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:19.136082 25773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:19.136155 25773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:19.136229 25773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:19.136281 25773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:19.145817 25773 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:41239
I20260926 12:41:19.145884 25901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:41239 every 8 connection(s)
I20260926 12:41:19.146895 25773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:41:19.152992 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 25773
Sep 26 12:41:19 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426479, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:19.162381 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.153789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.10.224.190:38441)
0926 12:41:19.154025 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:19.154030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:19.154239 (+ 209us) server_negotiation.cc:407] Connection header received
0926 12:41:19.154397 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:19.154400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:19.154452 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:19.154548 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:19.155356 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.156075 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.157044 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.157256 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.157334 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.158870 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:19.158877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:19.158880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:19.158939 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:19.160521 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:19.160914 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:19.160917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:19.160918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:19.160983 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:19.161321 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:19.161325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:19.161327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:19.161666 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:19.161865 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:19.162067 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 12:41:19.162186 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":163,"spinlock_wait_cycles":2304,"thread_start_us":96,"threads_started":1}
I20260926 12:41:19.162806 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.148858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:37145 (local address 127.10.224.190:38441)
0926 12:41:19.149033 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:19.149038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:19.149859 (+ 821us) server_negotiation.cc:407] Connection header received
0926 12:41:19.151188 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:19.151191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:19.151238 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:19.151314 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:19.152507 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.153037 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.154643 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.154873 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.155030 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.157954 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:19.157958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:19.157961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:19.157989 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:19.159798 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:19.160434 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:19.160439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:19.160441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:19.160521 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:19.161344 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:19.161346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:19.161348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:19.161656 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:19.161757 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:19.162524 (+ 767us) server_negotiation.cc:299] Negotiation successful
0926 12:41:19.162642 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":69}
I20260926 12:41:19.163811 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.149258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:37145)
0926 12:41:19.149707 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:19.149749 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:19.150990 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:19.151486 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:19.151492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:19.151896 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:19.152378 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.152391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.153165 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.153169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.154525 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.154534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.155069 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.155073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.155285 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.155991 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:19.156009 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:19.157788 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:19.159915 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:19.159923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:19.159944 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:19.160323 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:19.160944 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:19.160946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:19.160948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:19.161246 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:19.161796 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:19.161803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:19.162165 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 12:41:19.162484 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 12:41:19.162838 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260926 12:41:19.164932 25902 heartbeater.cc:347] Connected to a master server at 127.10.224.190:38441
I20260926 12:41:19.165136 25902 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:19.165575 25902 heartbeater.cc:511] Master 127.10.224.190:38441 requested a full tablet report, sending...
I20260926 12:41:19.166646 25444 ts_manager.cc:194] Registered new tserver with Master: 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239)
I20260926 12:41:19.167161 25444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:37145
I20260926 12:41:19.176393 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:41:19.180941 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.177087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.10.224.190:38441)
0926 12:41:19.177228 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:19.177232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:19.177398 (+ 166us) server_negotiation.cc:407] Connection header received
0926 12:41:19.177582 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:19.177584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:19.177704 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:19.177827 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:41:19.178086 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.178638 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.179630 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.179851 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.179932 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.180197 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:41:19.180615 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:41:19.180702 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:19.180770 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 12:41:19.180820 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:41:19.181636 25444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:41:19.185072 25444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:41:19.193343 25566 tablet_service.cc:1467] Processing CreateTablet for tablet f2abf9d31e0e446b961535c28a6ec14f (DEFAULT_TABLE table=test-table [id=c17098126c19497897a011b25be7776a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:19.194067 25566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2abf9d31e0e446b961535c28a6ec14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:19.197156 25524 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Bootstrap starting.
I20260926 12:41:19.197685 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.192009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:41239 (local address 127.0.0.1:34498)
0926 12:41:19.192151 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:19.192168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:19.192252 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:19.193212 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:19.193215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:19.193233 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:19.193360 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.193370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.194614 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.194621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.195874 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.195886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.197219 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.197224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.197348 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.197392 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 12:41:19.197443 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 12:41:19.197537 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260926 12:41:19.197934 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.192347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33779 (local address 127.0.0.1:48870)
0926 12:41:19.192693 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:19.192708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:19.192798 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:19.193841 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:19.193844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:19.193860 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:19.193952 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.193958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.195238 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.195243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.196268 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.196278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.197539 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.197542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.197623 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.197642 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:41:19.197720 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 12:41:19.197815 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
I20260926 12:41:19.198199 25524 tablet_bootstrap.cc:654] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:19.198362 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.192206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.10.224.131:41239)
0926 12:41:19.192606 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:19.192621 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:19.192662 (+ 41us) server_negotiation.cc:407] Connection header received
0926 12:41:19.192793 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:19.192800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:19.193051 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:19.193239 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:19.193464 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.194463 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.196010 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.197073 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.197882 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.198014 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:19.198070 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 12:41:19.198180 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":200,"threads_started":1}
I20260926 12:41:19.198378 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.192775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.10.224.130:33779)
0926 12:41:19.193216 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:19.193228 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:19.193250 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:41:19.193324 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:19.193329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:19.193522 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:19.193666 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:19.194081 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.195102 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.196409 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.197198 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.197334 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.197480 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:19.197974 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 12:41:19.198227 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":149,"threads_started":1}
I20260926 12:41:19.199723 25524 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: No bootstrap required, opened a new log
I20260926 12:41:19.199841 25524 ts_tablet_manager.cc:1397] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:19.200332 25524 raft_consensus.cc:348] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.200451 25524 raft_consensus.cc:374] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:19.200498 25524 raft_consensus.cc:729] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc27ac88a593467585e479e8e763cbaf, State: Initialized, Role: FOLLOWER
I20260926 12:41:19.200628 25524 consensus_queue.cc:260] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.200886 25524 ts_tablet_manager.cc:1428] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:41:19.201642 25704 tablet_service.cc:1467] Processing CreateTablet for tablet f2abf9d31e0e446b961535c28a6ec14f (DEFAULT_TABLE table=test-table [id=c17098126c19497897a011b25be7776a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:19.201999 25836 tablet_service.cc:1467] Processing CreateTablet for tablet f2abf9d31e0e446b961535c28a6ec14f (DEFAULT_TABLE table=test-table [id=c17098126c19497897a011b25be7776a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:19.202652 25704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2abf9d31e0e446b961535c28a6ec14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:19.203017 25836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2abf9d31e0e446b961535c28a6ec14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:19.209903 25916 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Bootstrap starting.
I20260926 12:41:19.209903 25915 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Bootstrap starting.
I20260926 12:41:19.212677 25915 tablet_bootstrap.cc:654] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:19.212689 25916 tablet_bootstrap.cc:654] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:19.213413 25915 log.cc:824] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:19.213413 25916 log.cc:824] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:19.215161 25916 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: No bootstrap required, opened a new log
I20260926 12:41:19.215168 25915 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: No bootstrap required, opened a new log
I20260926 12:41:19.215401 25916 ts_tablet_manager.cc:1397] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:41:19.215394 25915 ts_tablet_manager.cc:1397] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:41:19.218464 25916 raft_consensus.cc:348] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.218492 25915 raft_consensus.cc:348] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.218688 25916 raft_consensus.cc:374] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:19.218710 25915 raft_consensus.cc:374] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:19.218755 25916 raft_consensus.cc:729] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64ab430eba34be3bbda34f90f06d6f0, State: Initialized, Role: FOLLOWER
I20260926 12:41:19.218771 25915 raft_consensus.cc:729] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758872d19472487a85b61e8cbe991e61, State: Initialized, Role: FOLLOWER
I20260926 12:41:19.219185 25915 consensus_queue.cc:260] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.219161 25916 consensus_queue.cc:260] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.219930 25770 heartbeater.cc:503] Master 127.10.224.190:38441 was elected leader, sending a full tablet report...
I20260926 12:41:19.219930 25902 heartbeater.cc:503] Master 127.10.224.190:38441 was elected leader, sending a full tablet report...
I20260926 12:41:19.220273 25916 ts_tablet_manager.cc:1428] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:41:19.220274 25915 ts_tablet_manager.cc:1428] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 12:41:19.235956 25633 tablet.cc:2369] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:41:19.398023 25903 tablet.cc:2369] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:41:19.436650 25771 tablet.cc:2369] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:41:19.496031 25919 raft_consensus.cc:482] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:19.496240 25919 raft_consensus.cc:504] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.497433 25919 leader_election.cc:288] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779), cc27ac88a593467585e479e8e763cbaf (127.10.224.129:42237)
I20260926 12:41:19.501828 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.497926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:49917 (local address 127.10.224.130:33779)
0926 12:41:19.498048 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:19.498053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:19.498539 (+ 486us) server_negotiation.cc:407] Connection header received
0926 12:41:19.498594 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:19.498597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:19.498641 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:19.498746 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:19.498967 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.499710 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.500819 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.501263 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.501391 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.501445 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:19.501631 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 12:41:19.501701 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:41:19.501848 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.497821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33779 (local address 127.10.224.131:49917)
0926 12:41:19.498425 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:19.498437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:19.498520 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:19.498762 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:19.498764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:19.498778 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:19.498859 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.498863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.499843 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.499847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.500686 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.500695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.501388 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.501391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.501488 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.501517 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:41:19.501601 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:41:19.501679 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":202,"thread_start_us":180,"threads_started":1}
I20260926 12:41:19.501888 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.497732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42237 (local address 127.10.224.131:36853)
0926 12:41:19.497923 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:19.497937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:19.498036 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:19.498420 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:19.498423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:19.498447 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:19.498584 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.498591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.499733 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.499739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.500736 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:19.500745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.501474 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.501477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:19.501581 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.501600 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:41:19.501661 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 12:41:19.501732 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":11}
I20260926 12:41:19.501889 25921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.497827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:36853 (local address 127.10.224.129:42237)
0926 12:41:19.498098 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:19.498105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:19.498125 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:41:19.498185 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:19.498188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:19.498261 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:19.498382 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:19.498727 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.499596 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.500893 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.501347 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.501485 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.501653 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:19.501691 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 12:41:19.501758 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
I20260926 12:41:19.502660 25724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2abf9d31e0e446b961535c28a6ec14f" candidate_uuid: "758872d19472487a85b61e8cbe991e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64ab430eba34be3bbda34f90f06d6f0" is_pre_election: true
I20260926 12:41:19.502660 25586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2abf9d31e0e446b961535c28a6ec14f" candidate_uuid: "758872d19472487a85b61e8cbe991e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc27ac88a593467585e479e8e763cbaf" is_pre_election: true
I20260926 12:41:19.502960 25586 raft_consensus.cc:2454] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758872d19472487a85b61e8cbe991e61 in term 0.
I20260926 12:41:19.502970 25724 raft_consensus.cc:2454] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758872d19472487a85b61e8cbe991e61 in term 0.
I20260926 12:41:19.503505 25790 leader_election.cc:302] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758872d19472487a85b61e8cbe991e61, cc27ac88a593467585e479e8e763cbaf; no voters:
I20260926 12:41:19.503856 25919 raft_consensus.cc:2781] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:41:19.503957 25919 raft_consensus.cc:482] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:41:19.504016 25919 raft_consensus.cc:3037] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:19.505024 25919 raft_consensus.cc:504] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.505407 25919 leader_election.cc:288] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [CANDIDATE]: Term 1 election: Requested vote from peers b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779), cc27ac88a593467585e479e8e763cbaf (127.10.224.129:42237)
I20260926 12:41:19.505731 25724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2abf9d31e0e446b961535c28a6ec14f" candidate_uuid: "758872d19472487a85b61e8cbe991e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64ab430eba34be3bbda34f90f06d6f0"
I20260926 12:41:19.505816 25586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2abf9d31e0e446b961535c28a6ec14f" candidate_uuid: "758872d19472487a85b61e8cbe991e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc27ac88a593467585e479e8e763cbaf"
I20260926 12:41:19.505882 25724 raft_consensus.cc:3037] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:19.505947 25586 raft_consensus.cc:3037] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:19.507030 25586 raft_consensus.cc:2454] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758872d19472487a85b61e8cbe991e61 in term 1.
I20260926 12:41:19.507040 25724 raft_consensus.cc:2454] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758872d19472487a85b61e8cbe991e61 in term 1.
I20260926 12:41:19.507319 25790 leader_election.cc:302] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758872d19472487a85b61e8cbe991e61, cc27ac88a593467585e479e8e763cbaf; no voters:
I20260926 12:41:19.507647 25919 raft_consensus.cc:2781] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:19.507884 25919 raft_consensus.cc:686] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 1 LEADER]: Becoming Leader. State: Replica: 758872d19472487a85b61e8cbe991e61, State: Running, Role: LEADER
I20260926 12:41:19.508244 25919 consensus_queue.cc:237] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:19.511675 25443 catalog_manager.cc:5495] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 758872d19472487a85b61e8cbe991e61 (127.10.224.131). New cstate: current_term: 1 leader_uuid: "758872d19472487a85b61e8cbe991e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } health_report { overall_health: UNKNOWN } } }
I20260926 12:41:19.569432 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.565953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.10.224.131:41239)
0926 12:41:19.566089 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:19.566092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:19.566112 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:41:19.566198 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:19.566200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:19.566258 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:19.566353 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:41:19.566618 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.567120 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.567976 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:19.568248 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:19.568341 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:19.568632 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:41:19.569068 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:41:19.569180 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:19.569231 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 12:41:19.569295 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:41:19.575484 25724 raft_consensus.cc:1261] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 1 FOLLOWER]: Refusing update from remote peer 758872d19472487a85b61e8cbe991e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:19.575479 25586 raft_consensus.cc:1261] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Refusing update from remote peer 758872d19472487a85b61e8cbe991e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:19.576241 25923 consensus_queue.cc:1035] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:19.576450 25919 consensus_queue.cc:1035] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:19.583371 25927 mvcc.cc:205] Tried to move back new op lower bound from 7333586860333649920 to 7333586860068376576. Current Snapshot: MvccSnapshot[applied={T|T < 7333586860333649920}]
I20260926 12:41:19.585412 25928 mvcc.cc:205] Tried to move back new op lower bound from 7333586860333649920 to 7333586860068376576. Current Snapshot: MvccSnapshot[applied={T|T < 7333586860333649920}]
I20260926 12:41:19.591104 25443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:19.591288 25443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:19.593848 25443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Sending DeleteTablet for 3 replicas of tablet f2abf9d31e0e446b961535c28a6ec14f
I20260926 12:41:19.594583 25566 tablet_service.cc:1514] Processing DeleteTablet for tablet f2abf9d31e0e446b961535c28a6ec14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:19 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:33638
I20260926 12:41:19.594563 25836 tablet_service.cc:1514] Processing DeleteTablet for tablet f2abf9d31e0e446b961535c28a6ec14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:19 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:34498
I20260926 12:41:19.595005 25935 tablet_replica.cc:330] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: stopping tablet replica
I20260926 12:41:19.595026 25704 tablet_service.cc:1514] Processing DeleteTablet for tablet f2abf9d31e0e446b961535c28a6ec14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:19 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:48870
I20260926 12:41:19.595268 25935 raft_consensus.cc:2229] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:19.595422 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 25482
I20260926 12:41:19.595480 25937 tablet_replica.cc:330] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: stopping tablet replica
I20260926 12:41:19.596251 25937 raft_consensus.cc:2229] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:19.596570 25937 raft_consensus.cc:2258] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:19.598135 25937 ts_tablet_manager.cc:1905] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:19.601195 25937 ts_tablet_manager.cc:1918] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:19.601302 25937 log.cc:1196] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/f2abf9d31e0e446b961535c28a6ec14f
I20260926 12:41:19.601646 25937 ts_tablet_manager.cc:1939] T f2abf9d31e0e446b961535c28a6ec14f P b64ab430eba34be3bbda34f90f06d6f0: Deleting consensus metadata
I20260926 12:41:19.602983 25431 catalog_manager.cc:4841] TS b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779): tablet f2abf9d31e0e446b961535c28a6ec14f (table test-table [id=c17098126c19497897a011b25be7776a]) successfully deleted
W20260926 12:41:19.603220 25429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:42237 (error 108)
I20260926 12:41:19.603581 25936 tablet_replica.cc:330] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: stopping tablet replica
I20260926 12:41:19.603924 25936 raft_consensus.cc:2229] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:41:19.604296 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:19.603969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42237 (local address 127.0.0.1:33644)
0926 12:41:19.604144 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:19.604210 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:42237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 12:41:19.604329 25936 raft_consensus.cc:2258] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:41:19.604470 25429 catalog_manager.cc:4584] TS cc27ac88a593467585e479e8e763cbaf (127.10.224.129:42237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2abf9d31e0e446b961535c28a6ec14f: Network error: Client connection negotiation failed: client connection to 127.10.224.129:42237: connect: Connection refused (error 111)
I20260926 12:41:19.604715 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 25635
I20260926 12:41:19.608735 25936 ts_tablet_manager.cc:1905] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:19.611413 25936 ts_tablet_manager.cc:1918] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:19.611574 25936 log.cc:1196] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/f2abf9d31e0e446b961535c28a6ec14f
I20260926 12:41:19.611932 25936 ts_tablet_manager.cc:1939] T f2abf9d31e0e446b961535c28a6ec14f P 758872d19472487a85b61e8cbe991e61: Deleting consensus metadata
I20260926 12:41:19.613343 25428 catalog_manager.cc:4841] TS 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239): tablet f2abf9d31e0e446b961535c28a6ec14f (table test-table [id=c17098126c19497897a011b25be7776a]) successfully deleted
I20260926 12:41:19.613854 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 25773
I20260926 12:41:19.619233 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 25411
I20260926 12:41:19.625801 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38441
--webserver_interface=127.10.224.190
--webserver_port=40947
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:38441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:19.740744 25939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:19.741057 25939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:19.741102 25939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:19.745512 25939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:19.745594 25939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:19.745617 25939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:19.745635 25939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:19.745651 25939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:19.751376 25939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:38441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:38441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:19.752612 25939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:19.753690 25939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:19.762656 25944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:19.762698 25945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:19.762694 25939 server_base.cc:1034] running on GCE node
W20260926 12:41:19.762917 25947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:19.763622 25939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:19.764833 25939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:19.766000 25939 hybrid_clock.cc:648] HybridClock initialized: now 1790426479765982 us; error 34 us; skew 500 ppm
Sep 26 12:41:19 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426479, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:19.769093 25939 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:19.770069 25939 webserver.cc:460] Webserver started at http://127.10.224.190:40947/ using document root <none> and password file <none>
I20260926 12:41:19.770663 25939 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:19.770756 25939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:19.774947 25939 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:19.777387 25954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:19.778920 25939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:19.779105 25939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "69251a29accb4744b298e389fa937fd8"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "605e154befe9e2a9e77be92d90686b01"
server_key_iv: "9969a93a99da79640df46e44e5370cf8"
server_key_version: "encryptionkey@0"
I20260926 12:41:19.779793 25939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:19.801776 25939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:19.805121 25939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:19.805344 25939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:19.812749 25939 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:38441
I20260926 12:41:19.813024 26006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:38441 every 8 connection(s)
I20260926 12:41:19.813966 25939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:19.818574 26007 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:41:19.820634 26007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Bootstrap starting.
I20260926 12:41:19.822939 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 25939
I20260926 12:41:19.823450 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:42237
--local_ip_for_outbound_sockets=127.10.224.129
--tserver_master_addrs=127.10.224.190:38441
--webserver_port=46165
--webserver_interface=127.10.224.129
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:19.839488 26007 log.cc:824] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:19.851193 26007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:41:19.851696 26007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8: Bootstrap complete.
I20260926 12:41:19.855782 26007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:19.856191 26007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69251a29accb4744b298e389fa937fd8, State: Initialized, Role: FOLLOWER
I20260926 12:41:19.856756 26007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:19.856882 26007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:41:19.856968 26007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:41:19.857087 26007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:41:19.859140 26007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:19.859494 26007 leader_election.cc:302] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69251a29accb4744b298e389fa937fd8; no voters:
I20260926 12:41:19.859889 26007 leader_election.cc:288] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 12:41:19.859980 26011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:41:19.860191 26011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [term 3 LEADER]: Becoming Leader. State: Replica: 69251a29accb4744b298e389fa937fd8, State: Running, Role: LEADER
I20260926 12:41:19.860489 26011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } }
I20260926 12:41:19.861153 26007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:19.861804 26012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "69251a29accb4744b298e389fa937fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } } }
I20260926 12:41:19.861930 26012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:19.862633 26018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:19.862663 26013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69251a29accb4744b298e389fa937fd8. Latest consensus state: current_term: 3 leader_uuid: "69251a29accb4744b298e389fa937fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69251a29accb4744b298e389fa937fd8" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 38441 } } }
I20260926 12:41:19.862782 26013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69251a29accb4744b298e389fa937fd8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:19.868028 26018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:19.868819 26018 catalog_manager.cc:1259] Loaded cluster ID: 327f86d5c51f45a0b0bf9b06e079be19
I20260926 12:41:19.868906 26018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 12:41:19.870266 26018 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd9ff8789ad google::LogMessage::Fail() at ??:0
@ 0x7fd9ff87b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd9ff8784de google::LogMessage::Flush() at ??:0
@ 0x7fd9ff879a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fda0a1b7fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fda0a1b8d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fda0a1b5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fda0a1f9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fda01aca3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fda01acac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fda01acc3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fda01abb512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fda030ff6db start_thread at ??:0
@ 0x7fd9fe37661f clone at ??:0
W20260926 12:41:19.981926 26009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:19.982419 26009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:19.982496 26009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:19.989481 26009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:19.989596 26009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:19.989701 26009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:41:19.998241 26009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:42237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=46165
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:19.999958 26009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:20.001588 26009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:20.014008 26051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:20.013970 26048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:20.013996 26049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:20.014125 26009 server_base.cc:1034] running on GCE node
I20260926 12:41:20.014660 26009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:20.015374 26009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:20.016574 26009 hybrid_clock.cc:648] HybridClock initialized: now 1790426480016507 us; error 86 us; skew 500 ppm
Sep 26 12:41:20 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:20.019635 26009 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:41:20.020684 26009 webserver.cc:460] Webserver started at http://127.10.224.129:46165/ using document root <none> and password file <none>
I20260926 12:41:20.021276 26009 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:20.021340 26009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:20.025832 26009 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:41:20.028370 26058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:20.029740 26009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:20.029886 26009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "cc27ac88a593467585e479e8e763cbaf"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "633288bef6e16d00844162a90d3d9100"
server_key_iv: "ad59d64b845ccbfff10decd51936d247"
server_key_version: "encryptionkey@0"
I20260926 12:41:20.030454 26009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:20.048806 26009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:20.052493 26009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:20.052718 26009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:20.053335 26009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:20.054499 26065 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:41:20.060277 26009 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:41:20.060375 26009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 12:41:20.060479 26009 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:41:20.062330 26009 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:41:20.062407 26009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:41:20.062762 26065 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Bootstrap starting.
I20260926 12:41:20.073032 26009 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:42237
I20260926 12:41:20.073297 26172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:42237 every 8 connection(s)
I20260926 12:41:20.074186 26009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:41:20.081643 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 26009
I20260926 12:41:20.082125 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:33779
--local_ip_for_outbound_sockets=127.10.224.130
--tserver_master_addrs=127.10.224.190:38441
--webserver_port=40455
--webserver_interface=127.10.224.130
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:20.086727 26065 log.cc:824] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:20.096609 26065 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:41:20.097074 26065 tablet_bootstrap.cc:492] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Bootstrap complete.
I20260926 12:41:20.098572 26065 ts_tablet_manager.cc:1397] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.008s
I20260926 12:41:20.102324 26065 raft_consensus.cc:348] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:20.102905 26065 raft_consensus.cc:729] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc27ac88a593467585e479e8e763cbaf, State: Initialized, Role: FOLLOWER
I20260926 12:41:20.103483 26065 consensus_queue.cc:260] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:20.104676 26065 ts_tablet_manager.cc:1428] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf: Time spent starting tablet: real 0.006s user 0.000s sys 0.008s
W20260926 12:41:20.159276 26175 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:41:20.076298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:40713)
0926 12:41:20.076807 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.076865 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:20.077751 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:20.158244 (+ 80493us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":160,"threads_started":1}
W20260926 12:41:20.159915 26173 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:38441 (0 consecutive failures): Network error: Failed to ping master at 127.10.224.190:38441: Client connection negotiation failed: client connection to 127.10.224.190:38441: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:41:20.160717 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:20.160409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:55643)
0926 12:41:20.160568 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.160647 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 12:41:20.161324 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:20.161109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:52997)
0926 12:41:20.161225 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.161272 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:41:20.161518 26173 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:41:20.202360 26176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:20.202641 26176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:20.202694 26176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:20.206795 26176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:20.206873 26176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:20.206934 26176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:41:20.212453 26176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:33779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=40455
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:20.213486 26176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:20.214517 26176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:41:20.224129 26176 server_base.cc:1034] running on GCE node
W20260926 12:41:20.224044 26184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:20.224061 26183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:20.224061 26186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:20.224785 26176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:20.225476 26176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:20.226660 26176 hybrid_clock.cc:648] HybridClock initialized: now 1790426480226644 us; error 29 us; skew 500 ppm
Sep 26 12:41:20 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:20.229382 26176 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:41:20.230270 26176 webserver.cc:460] Webserver started at http://127.10.224.130:40455/ using document root <none> and password file <none>
I20260926 12:41:20.230813 26176 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:20.230875 26176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:20.235118 26176 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:20.237358 26193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:20.238713 26176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:41:20.238848 26176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "b64ab430eba34be3bbda34f90f06d6f0"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "07a801309d491a3bf2632701f6f8d315"
server_key_iv: "f243bc71ab0db919c35c89bbecac8991"
server_key_version: "encryptionkey@0"
I20260926 12:41:20.239424 26176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:20.255776 26176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:20.258757 26176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:20.258971 26176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:20.259647 26176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:20.260511 26176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:20.260557 26176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:20.260629 26176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:20.260679 26176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:20.270181 26176 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:33779
I20260926 12:41:20.270201 26306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:33779 every 8 connection(s)
I20260926 12:41:20.271277 26176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:41:20.271921 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 26176
I20260926 12:41:20.272398 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:41239
--local_ip_for_outbound_sockets=127.10.224.131
--tserver_master_addrs=127.10.224.190:38441
--webserver_port=36689
--webserver_interface=127.10.224.131
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:20.275815 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:20.273591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:42763)
0926 12:41:20.274141 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.274757 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":113,"threads_started":1}
W20260926 12:41:20.276291 26307 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:38441 (0 consecutive failures): Network error: Failed to ping master at 127.10.224.190:38441: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
I20260926 12:41:20.276911 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:20.276676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:42065)
0926 12:41:20.276802 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.276859 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 12:41:20.277558 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:20.277352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:56647)
0926 12:41:20.277459 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.277502 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:41:20.277706 26307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:41:20.390944 26309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:20.391350 26309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:20.391445 26309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:20.395892 26309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:20.396011 26309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:20.396118 26309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:41:20.402050 26309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:41239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=36689
--tserver_master_addrs=127.10.224.190:38441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:20.403333 26309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:20.404529 26309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:20.415038 26318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:20.415109 26309 server_base.cc:1034] running on GCE node
W20260926 12:41:20.415043 26316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:20.415077 26315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:20.415875 26309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:20.416529 26309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:20.417695 26309 hybrid_clock.cc:648] HybridClock initialized: now 1790426480417663 us; error 47 us; skew 500 ppm
Sep 26 12:41:20 dist-test-slave-m3pj krb5kdc[24863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:20.421072 26309 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:41:20.422237 26309 webserver.cc:460] Webserver started at http://127.10.224.131:36689/ using document root <none> and password file <none>
I20260926 12:41:20.422878 26309 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:20.422951 26309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:20.427311 26309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:41:20.429831 26325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:20.431324 26309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:20.431497 26309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "758872d19472487a85b61e8cbe991e61"
format_stamp: "Formatted at 2026-09-26 12:41:17 on dist-test-slave-m3pj"
server_key: "858da58ab1166a04ffa2952e88023f15"
server_key_iv: "62e97e85ffdc88bd3825ff39351a28fa"
server_key_version: "encryptionkey@0"
I20260926 12:41:20.432179 26309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:20.473278 26309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:20.476845 26309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:20.477079 26309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:20.477733 26309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:20.478629 26309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:20.478705 26309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:20.478775 26309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:20.478828 26309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:20.488930 26309 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:41239
I20260926 12:41:20.488967 26438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:41239 every 8 connection(s)
I20260926 12:41:20.490016 26309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:41:20.490834 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 26309
I20260926 12:41:20.494230 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:20.492431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:46013)
0926 12:41:20.492884 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.492999 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
W20260926 12:41:20.494777 26439 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:38441 (0 consecutive failures): Network error: Failed to ping master at 127.10.224.190:38441: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
I20260926 12:41:20.495357 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:20.495119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:37405)
0926 12:41:20.495242 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.495300 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 12:41:20.495966 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:20.495776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:47207)
0926 12:41:20.495874 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:20.495915 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 12:41:20.496133 26439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 12:41:21.162744 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:21.162233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:43203)
0926 12:41:21.162538 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:21.162621 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260926 12:41:21.278812 26444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:21.278345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:40633)
0926 12:41:21.278651 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:21.278706 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
I20260926 12:41:21.461493 26445 raft_consensus.cc:482] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:21.461771 26445 raft_consensus.cc:504] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:21.463001 26445 leader_election.cc:288] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239), b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
Sep 26 12:41:21 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.131@KRBTEST.COM
Sep 26 12:41:21 dist-test-slave-m3pj krb5kdc[24863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426480, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.130@KRBTEST.COM
I20260926 12:41:21.480558 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:21.463597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:39017 (local address 127.10.224.131:41239)
0926 12:41:21.464058 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:21.464075 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:21.464125 (+ 50us) server_negotiation.cc:407] Connection header received
0926 12:41:21.464773 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:21.464780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:21.465132 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:21.465459 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:21.466723 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:21.468508 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:21.470655 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:21.471404 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:21.471609 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:21.475322 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:21.475328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:21.475342 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:21.475371 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:21.477687 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:21.478414 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:21.478421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:21.478428 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:21.478557 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:21.478936 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:21.478940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:21.478942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:21.479383 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:21.479554 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:21.480089 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 12:41:21.480365 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":111,"threads_started":1}
I20260926 12:41:21.480548 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:21.463337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:41239 (local address 127.10.224.129:39017)
0926 12:41:21.463875 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:21.463889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:21.463986 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:21.465506 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:21.465512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:21.466069 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:21.466534 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:21.466552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:21.468713 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:21.468717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:21.470507 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:21.470520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:21.471606 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:21.471619 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:21.471934 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:21.472829 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:21.472865 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:21.475054 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:21.477829 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:21.477838 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:21.477857 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:21.478282 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:21.478659 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:21.478663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:21.478665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:21.478835 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:21.479502 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:21.479509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:21.479775 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 12:41:21.479975 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 12:41:21.480339 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":491,"spinlock_wait_cycles":95232,"thread_start_us":59,"threads_started":1}
I20260926 12:41:21.480592 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:21.463821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:46145 (local address 127.10.224.130:33779)
0926 12:41:21.464099 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:21.464121 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:21.464154 (+ 33us) server_negotiation.cc:407] Connection header received
0926 12:41:21.464832 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:21.464838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:21.465132 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:21.465460 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:21.466772 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:21.468508 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:21.470655 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:21.471404 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:21.471609 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:21.475414 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:21.475419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:21.475430 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:21.475459 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:21.477694 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:21.478413 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:21.478421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:21.478431 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:21.478555 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:21.478950 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:21.478953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:21.478955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:21.479382 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:21.479551 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:21.480105 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 12:41:21.480390 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260926 12:41:21.480602 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:21.463337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33779 (local address 127.10.224.129:46145)
0926 12:41:21.463448 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:21.463465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:21.463602 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:21.465506 (+ 1904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:21.465512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:21.466069 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:21.466536 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:21.466552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:21.468713 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:21.468717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:21.470506 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:21.470520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:21.471606 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:21.471611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:21.471937 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:21.472829 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:21.472860 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:21.475197 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:21.477839 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:21.477842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:21.477857 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:21.478286 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:21.478658 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:21.478661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:21.478663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:21.478842 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:21.479502 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:21.479509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:21.479775 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 12:41:21.479978 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 12:41:21.480422 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":521,"spinlock_wait_cycles":39424}
W20260926 12:41:21.483598 26059 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
W20260926 12:41:21.483824 26062 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
I20260926 12:41:21.483937 26062 leader_election.cc:302] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc27ac88a593467585e479e8e763cbaf; no voters: 758872d19472487a85b61e8cbe991e61, b64ab430eba34be3bbda34f90f06d6f0
I20260926 12:41:21.484275 26445 raft_consensus.cc:2726] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:41:21.497195 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:21.496739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:42037)
0926 12:41:21.497015 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:21.497080 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260926 12:41:22.164275 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:22.163781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:51535)
0926 12:41:22.164066 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:22.164133 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":81,"threads_started":1}
I20260926 12:41:22.280205 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:22.279775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:45387)
0926 12:41:22.280045 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:22.280108 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260926 12:41:22.498389 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:22.497959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:60883)
0926 12:41:22.498217 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:22.498277 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260926 12:41:23.030979 26453 raft_consensus.cc:482] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:23.031109 26453 raft_consensus.cc:504] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:23.031661 26453 leader_election.cc:288] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239), b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
W20260926 12:41:23.032539 26062 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
W20260926 12:41:23.032651 26059 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
I20260926 12:41:23.032780 26059 leader_election.cc:302] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc27ac88a593467585e479e8e763cbaf; no voters: 758872d19472487a85b61e8cbe991e61, b64ab430eba34be3bbda34f90f06d6f0
I20260926 12:41:23.032976 26453 raft_consensus.cc:2726] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:41:23.165720 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:23.165236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:40563)
0926 12:41:23.165552 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:23.165619 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260926 12:41:23.281582 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:23.281036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:58775)
0926 12:41:23.281384 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:23.281454 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
I20260926 12:41:23.499897 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:23.499301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:57485)
0926 12:41:23.499665 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:23.499729 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260926 12:41:24.167227 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:24.166709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:50077)
0926 12:41:24.166995 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:24.167058 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260926 12:41:24.282960 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:24.282501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:60315)
0926 12:41:24.282798 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:24.282861 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260926 12:41:24.501091 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:24.500685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:40515)
0926 12:41:24.500940 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:24.500993 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260926 12:41:24.982445 26460 raft_consensus.cc:482] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:24.982620 26460 raft_consensus.cc:504] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:24.983037 26460 leader_election.cc:288] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239), b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
W20260926 12:41:24.984088 26062 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
W20260926 12:41:24.984261 26059 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
I20260926 12:41:24.984336 26059 leader_election.cc:302] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc27ac88a593467585e479e8e763cbaf; no voters: 758872d19472487a85b61e8cbe991e61, b64ab430eba34be3bbda34f90f06d6f0
I20260926 12:41:24.984494 26460 raft_consensus.cc:2726] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:41:25.168706 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:25.168214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:41341)
0926 12:41:25.168543 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:25.168606 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":90,"threads_started":1}
I20260926 12:41:25.284600 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:25.284061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:48025)
0926 12:41:25.284436 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:25.284504 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260926 12:41:25.502570 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:25.501993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:39709)
0926 12:41:25.502337 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:25.502409 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260926 12:41:26.170344 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:26.169802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:50473)
0926 12:41:26.170116 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:26.170241 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":83,"threads_started":1}
I20260926 12:41:26.285967 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:26.285509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:41889)
0926 12:41:26.285777 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:26.285837 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260926 12:41:26.504087 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:26.503634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:57007)
0926 12:41:26.503885 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:26.503954 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":111,"threads_started":1}
I20260926 12:41:27.171809 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:27.171245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:44239)
0926 12:41:27.171626 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:27.171707 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260926 12:41:27.287685 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:27.286941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:56631)
0926 12:41:27.287440 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:27.287500 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":162,"threads_started":1}
I20260926 12:41:27.505457 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:27.504974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:34023)
0926 12:41:27.505292 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:27.505356 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
I20260926 12:41:27.519508 26470 raft_consensus.cc:482] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:27.519706 26470 raft_consensus.cc:504] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:27.520186 26470 leader_election.cc:288] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239), b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
W20260926 12:41:27.520988 26059 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
W20260926 12:41:27.521075 26062 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
I20260926 12:41:27.521181 26062 leader_election.cc:302] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc27ac88a593467585e479e8e763cbaf; no voters: 758872d19472487a85b61e8cbe991e61, b64ab430eba34be3bbda34f90f06d6f0
I20260926 12:41:27.521323 26470 raft_consensus.cc:2726] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:41:28.173175 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:28.172692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:51173)
0926 12:41:28.172998 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:28.173070 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260926 12:41:28.288945 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:28.288522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:60275)
0926 12:41:28.288779 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:28.288840 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260926 12:41:28.506610 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:28.506228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:47943)
0926 12:41:28.506462 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:28.506514 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260926 12:41:29.174504 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:29.174082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:53695)
0926 12:41:29.174342 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:29.174404 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260926 12:41:29.290104 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:29.289702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:48519)
0926 12:41:29.289950 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:29.290007 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260926 12:41:29.351992 26476 raft_consensus.cc:482] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:29.352171 26476 raft_consensus.cc:504] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758872d19472487a85b61e8cbe991e61" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 41239 } } peers { permanent_uuid: "b64ab430eba34be3bbda34f90f06d6f0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33779 } } peers { permanent_uuid: "cc27ac88a593467585e479e8e763cbaf" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42237 } }
I20260926 12:41:29.352677 26476 leader_election.cc:288] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239), b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779)
W20260926 12:41:29.353586 26059 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758872d19472487a85b61e8cbe991e61 (127.10.224.131:41239): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
W20260926 12:41:29.353688 26062 leader_election.cc:341] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b64ab430eba34be3bbda34f90f06d6f0 (127.10.224.130:33779): Not found: Tablet not found: f2abf9d31e0e446b961535c28a6ec14f
I20260926 12:41:29.353760 26062 leader_election.cc:302] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc27ac88a593467585e479e8e763cbaf; no voters: 758872d19472487a85b61e8cbe991e61, b64ab430eba34be3bbda34f90f06d6f0
I20260926 12:41:29.353916 26476 raft_consensus.cc:2726] T f2abf9d31e0e446b961535c28a6ec14f P cc27ac88a593467585e479e8e763cbaf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:41:29.508065 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:29.507514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:57021)
0926 12:41:29.507892 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:29.507964 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20260926 12:41:30.176118 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:30.175658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:49963)
0926 12:41:30.175943 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:30.176011 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260926 12:41:30.291677 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:30.291087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:48477)
0926 12:41:30.291425 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:30.291492 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260926 12:41:30.509435 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:30.509001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:39723)
0926 12:41:30.509270 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:30.509335 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260926 12:41:31.177524 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:31.177057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:42259)
0926 12:41:31.177356 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:31.177422 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260926 12:41:31.292922 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:31.292485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:58055)
0926 12:41:31.292759 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:31.292820 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260926 12:41:31.510754 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:31.510283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:32855)
0926 12:41:31.510588 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:31.510652 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260926 12:41:32.178808 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:32.178409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:55997)
0926 12:41:32.178653 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:32.178713 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260926 12:41:32.294377 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:32.293885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:35437)
0926 12:41:32.294205 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:32.294274 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
I20260926 12:41:32.512025 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:32.511598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:56577)
0926 12:41:32.511856 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:32.511915 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260926 12:41:33.180270 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:33.179772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:55819)
0926 12:41:33.180101 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:33.180167 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":113,"threads_started":1}
I20260926 12:41:33.295868 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:33.295345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:41977)
0926 12:41:33.295696 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:33.295764 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":85,"threads_started":1}
I20260926 12:41:33.513393 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:33.512915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:59443)
0926 12:41:33.513216 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:33.513289 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
I20260926 12:41:34.181499 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:34.181065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:40257)
0926 12:41:34.181348 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:34.181404 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20260926 12:41:34.297209 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:34.296753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:45043)
0926 12:41:34.297039 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:34.297104 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260926 12:41:34.514772 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:34.514303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.131:56559)
0926 12:41:34.514587 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:34.514653 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260926 12:41:35.183116 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:35.182628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.129:36183)
0926 12:41:35.182930 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:35.183012 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
I20260926 12:41:35.298718 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:35.298267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:38441 (local address 127.10.224.130:60463)
0926 12:41:35.298553 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:35.298616 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260926 12:41:35.494016 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 26009
I20260926 12:41:35.499946 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 26176
I20260926 12:41:35.505841 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 26309
2026-09-26T12:41:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19708 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:41:35 dist-test-slave-m3pj krb5kdc[26497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:41:35 dist-test-slave-m3pj krb5kdc[26497](info): set up 2 sockets
Sep 26 12:41:35 dist-test-slave-m3pj krb5kdc[26497](info): commencing operation
krb5kdc: starting...
W20260926 12:41:36.560281 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:41:36 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:41:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:41:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:41:36.687840 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34831
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:34831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:36.799381 26510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:36.799754 26510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:36.799865 26510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:36.803959 26510 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:41:36.804033 26510 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:41:36.804061 26510 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:41:36.804095 26510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:36.804134 26510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:36.804179 26510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:36.804212 26510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:36.804248 26510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:36.809587 26510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:34831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:36.810725 26510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:36.811877 26510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:36.821166 26516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:36.821162 26518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:36.821619 26515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:36.821712 26510 server_base.cc:1034] running on GCE node
I20260926 12:41:36.822294 26510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:36.823494 26510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:36.824712 26510 hybrid_clock.cc:648] HybridClock initialized: now 1790426496824698 us; error 26 us; skew 500 ppm
Sep 26 12:41:36 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426496, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:36.827587 26510 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:36.828483 26510 webserver.cc:460] Webserver started at http://127.10.224.190:35669/ using document root <none> and password file <none>
I20260926 12:41:36.829005 26510 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:36.829064 26510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:36.829231 26510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:36.831213 26510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "34c53f617429429d8671bae159514186"
format_stamp: "Formatted at 2026-09-26 12:41:36 on dist-test-slave-m3pj"
server_key: "ef3adbed4ecb44695a7a9ec9ddce6e16"
server_key_iv: "13d13ce09218cdf4923f3aa251243719"
server_key_version: "encryptionkey@0"
I20260926 12:41:36.831779 26510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "34c53f617429429d8671bae159514186"
format_stamp: "Formatted at 2026-09-26 12:41:36 on dist-test-slave-m3pj"
server_key: "ef3adbed4ecb44695a7a9ec9ddce6e16"
server_key_iv: "13d13ce09218cdf4923f3aa251243719"
server_key_version: "encryptionkey@0"
I20260926 12:41:36.835362 26510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:41:36.837715 26525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:36.838917 26510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:41:36.839046 26510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "34c53f617429429d8671bae159514186"
format_stamp: "Formatted at 2026-09-26 12:41:36 on dist-test-slave-m3pj"
server_key: "ef3adbed4ecb44695a7a9ec9ddce6e16"
server_key_iv: "13d13ce09218cdf4923f3aa251243719"
server_key_version: "encryptionkey@0"
I20260926 12:41:36.839150 26510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:36.865828 26510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:36.868780 26510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:36.868985 26510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:36.875900 26510 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:34831
I20260926 12:41:36.876191 26577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:34831 every 8 connection(s)
I20260926 12:41:36.876895 26510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:36.880152 26578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:36.884644 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 26510
I20260926 12:41:36.884792 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:41:36.885073 11138 external_mini_cluster.cc:1428] Setting key c510f1c764e16e437050b4e3f7e4443c
I20260926 12:41:36.886173 26578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Bootstrap starting.
I20260926 12:41:36.888664 26578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:36.889427 26578 log.cc:824] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:36.891372 26578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: No bootstrap required, opened a new log
Sep 26 12:41:36 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:36.895352 26578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:41:36.895687 26578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:36.895785 26578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c53f617429429d8671bae159514186, State: Initialized, Role: FOLLOWER
I20260926 12:41:36.896291 26578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:41:36.896457 26578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:41:36.896541 26578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:41:36.896658 26578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:36.897600 26578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:41:36.897953 26578 leader_election.cc:302] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34c53f617429429d8671bae159514186; no voters:
I20260926 12:41:36.898283 26578 leader_election.cc:288] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:41:36.898674 26584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:36.898909 26584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 1 LEADER]: Becoming Leader. State: Replica: 34c53f617429429d8671bae159514186, State: Running, Role: LEADER
I20260926 12:41:36.899266 26584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:41:36.899711 26578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:36.900183 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:36.886646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.10.224.190:34831)
0926 12:41:36.887094 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:36.887109 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:36.887144 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:41:36.887880 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:36.887899 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:36.888244 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:36.888599 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:36.889404 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:36.890464 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:36.891315 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:36.891915 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:36.892087 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:36.894667 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:36.894672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:36.894688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:36.894726 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:36.896769 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:36.897250 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:36.897257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:36.897263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:36.897385 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:36.897677 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:36.897681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:36.897683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:36.898190 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:36.898398 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:36.898713 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 12:41:36.899194 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":138,"threads_started":1}
I20260926 12:41:36.901997 26583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34c53f617429429d8671bae159514186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } } }
I20260926 12:41:36.903231 26583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:36.902864 26585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34c53f617429429d8671bae159514186. Latest consensus state: current_term: 1 leader_uuid: "34c53f617429429d8671bae159514186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } } }
I20260926 12:41:36.903621 26585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: This master's current role is: LEADER
W20260926 12:41:36.904611 26597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:41:36.904702 26597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:41:36.904951 26600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:36.905670 26600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:36.911226 26600 catalog_manager.cc:1347] Generated new cluster ID: df17f2acd7834052a64935ddd6011598
I20260926 12:41:36.911295 26600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:41:36.936741 26600 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:41:36.937847 26600 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:41:36.943773 26600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Generated new TSK 0
I20260926 12:41:36.944602 26600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:41:36.983371 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34831
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:37.102016 26604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:37.102381 26604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:37.102468 26604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:37.106881 26604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:37.106993 26604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:37.107095 26604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:41:37.112989 26604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:37.114207 26604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:37.115365 26604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:37.125741 26612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:37.125761 26610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:37.125769 26609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:37.125916 26604 server_base.cc:1034] running on GCE node
I20260926 12:41:37.126561 26604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:37.127285 26604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:37.128475 26604 hybrid_clock.cc:648] HybridClock initialized: now 1790426497128444 us; error 44 us; skew 500 ppm
Sep 26 12:41:37 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:37.131618 26604 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:41:37.132668 26604 webserver.cc:460] Webserver started at http://127.10.224.129:33163/ using document root <none> and password file <none>
I20260926 12:41:37.133293 26604 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:37.133369 26604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:37.133615 26604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:37.135931 26604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "9422c84e0487a737454548502d20cadc"
server_key_iv: "573c40b711a7215663e5c7b7b3d740a3"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.136523 26604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "9422c84e0487a737454548502d20cadc"
server_key_iv: "573c40b711a7215663e5c7b7b3d740a3"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.140949 26604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:41:37.143859 26619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.145244 26604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:41:37.145426 26604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "9422c84e0487a737454548502d20cadc"
server_key_iv: "573c40b711a7215663e5c7b7b3d740a3"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.145565 26604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:37.159368 26604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:37.163003 26604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:37.163225 26604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:37.163928 26604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:37.164800 26604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:37.164872 26604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.164945 26604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:37.164987 26604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.174024 26604 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:39179
I20260926 12:41:37.174047 26732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:39179 every 8 connection(s)
I20260926 12:41:37.174953 26604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:41:37.181166 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 26604
I20260926 12:41:37.181300 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:41:37.181605 11138 external_mini_cluster.cc:1428] Setting key be08e2642ead8d1d6f6f627a070ae0f6
Sep 26 12:41:37 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:37.189261 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.176813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:59915 (local address 127.10.224.190:34831)
0926 12:41:37.176961 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.176965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.177778 (+ 813us) server_negotiation.cc:407] Connection header received
0926 12:41:37.178989 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.178992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.179040 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.179116 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:37.180291 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.180848 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.182443 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.182693 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.182825 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.185483 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:37.185486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:37.185488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:37.185512 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:37.187088 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:37.187644 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:37.187650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:37.187655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:37.187741 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:37.188095 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:37.188098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:37.188100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:37.188299 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:37.188408 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.188931 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.189064 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:41:37.190065 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.177191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:59915)
0926 12:41:37.177625 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.177668 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:37.178792 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:37.179265 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:37.179272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:37.179675 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:37.180152 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.180166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.180984 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.180988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.182293 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.182313 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.182875 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.182879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.183067 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.183778 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:37.183794 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:37.185306 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:37.187221 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:37.187227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:37.187242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:37.187507 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:37.187841 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:37.187844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:37.187846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:37.187990 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:37.188409 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:37.188417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:37.188685 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 12:41:37.188876 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 12:41:37.189205 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:41:37.191386 26733 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34831
I20260926 12:41:37.191654 26733 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:37.192207 26733 heartbeater.cc:511] Master 127.10.224.190:34831 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:41:37.193783 26542 ts_manager.cc:194] Registered new tserver with Master: 9c7e1a2af6fa4922ae0586af5dd4ac2d (127.10.224.129:39179)
I20260926 12:41:37.195021 26542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:59915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:41:37.210788 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34831
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:37.327488 26738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:37.327785 26738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:37.327831 26738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:37.331930 26738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:37.331995 26738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:37.332105 26738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:41:37.337903 26738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:37.339061 26738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:37.340255 26738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:37.350726 26744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:37.350750 26743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:37.350706 26746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:37.350965 26738 server_base.cc:1034] running on GCE node
I20260926 12:41:37.351419 26738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:37.352180 26738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:37.353351 26738 hybrid_clock.cc:648] HybridClock initialized: now 1790426497353314 us; error 49 us; skew 500 ppm
Sep 26 12:41:37 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:37.356256 26738 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:41:37.357251 26738 webserver.cc:460] Webserver started at http://127.10.224.130:39953/ using document root <none> and password file <none>
I20260926 12:41:37.357806 26738 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:37.357880 26738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:37.358055 26738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:37.360261 26738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "2654cb4c5fe44695bec5770bd3585be5"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "3205e7694c835876a041fc4b14f3806a"
server_key_iv: "049116ad16398af7388f29efce9e2e08"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.360811 26738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "2654cb4c5fe44695bec5770bd3585be5"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "3205e7694c835876a041fc4b14f3806a"
server_key_iv: "049116ad16398af7388f29efce9e2e08"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.364977 26738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:41:37.367697 26753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.369076 26738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:41:37.369226 26738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "2654cb4c5fe44695bec5770bd3585be5"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "3205e7694c835876a041fc4b14f3806a"
server_key_iv: "049116ad16398af7388f29efce9e2e08"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.369364 26738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:37.393898 26738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:37.397413 26738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:37.397627 26738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:37.398241 26738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:37.399209 26738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:37.399279 26738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.399381 26738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:37.399410 26738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.409446 26738 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:41695
I20260926 12:41:37.409466 26866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:41695 every 8 connection(s)
I20260926 12:41:37.410483 26738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:41:37.417853 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 26738
I20260926 12:41:37.418008 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:41:37.418305 11138 external_mini_cluster.cc:1428] Setting key 182fcd4366a9725c8a6bd6613ed9aa40
Sep 26 12:41:37 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:37.425030 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.412605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:37359 (local address 127.10.224.190:34831)
0926 12:41:37.412774 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.412778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.414014 (+ 1236us) server_negotiation.cc:407] Connection header received
0926 12:41:37.414867 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.414870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.414917 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.414997 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:37.416257 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.416788 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.417931 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.418265 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.418380 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.421070 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:37.421076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:37.421082 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:37.421116 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:37.422615 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:37.423240 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:37.423242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:37.423244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:37.423316 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:37.423745 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:37.423747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:37.423749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:37.423979 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:37.424087 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.424778 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.424902 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:41:37.426403 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.413135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:37359)
0926 12:41:37.413844 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.413886 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:37.414640 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:37.415164 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:37.415170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:37.415624 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:37.416124 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.416138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.416907 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.416910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.417814 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.417823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.418452 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.418459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.418613 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.419240 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:37.419257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:37.420897 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:37.422736 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:37.422745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:37.422765 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:37.423142 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:37.423408 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:37.423415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:37.423420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:37.423642 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:37.424113 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:37.424124 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:37.424499 (+ 375us) client_negotiation.cc:769] Sending connection context
0926 12:41:37.424717 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 12:41:37.425130 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":119,"threads_started":1}
I20260926 12:41:37.427790 26867 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34831
I20260926 12:41:37.428054 26867 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:37.428586 26867 heartbeater.cc:511] Master 127.10.224.190:34831 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:41:37.429726 26542 ts_manager.cc:194] Registered new tserver with Master: 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
I20260926 12:41:37.430330 26542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:37359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:41:37.446836 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34831
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:37.562182 26872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:37.562530 26872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:37.562575 26872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:37.566629 26872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:37.566694 26872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:37.566754 26872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:41:37.572088 26872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:37.573237 26872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:37.574275 26872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:37.584324 26880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:37.584414 26872 server_base.cc:1034] running on GCE node
W20260926 12:41:37.584345 26878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:37.584345 26877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:37.585019 26872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:37.585784 26872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:37.587010 26872 hybrid_clock.cc:648] HybridClock initialized: now 1790426497586970 us; error 49 us; skew 500 ppm
Sep 26 12:41:37 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:37.590006 26872 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:41:37.591014 26872 webserver.cc:460] Webserver started at http://127.10.224.131:35681/ using document root <none> and password file <none>
I20260926 12:41:37.591694 26872 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:37.591790 26872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:37.592017 26872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:41:37.594513 26872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "17fe564b62434feead95d2ec809e166c"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "af5d1e11825fe25c203b9be716fd91de"
server_key_iv: "e77ac0fe09392941d91755afaa1b6f40"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.595118 26872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "17fe564b62434feead95d2ec809e166c"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "af5d1e11825fe25c203b9be716fd91de"
server_key_iv: "e77ac0fe09392941d91755afaa1b6f40"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.599150 26872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:41:37.602070 26887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.603439 26872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:37.603616 26872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "17fe564b62434feead95d2ec809e166c"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "af5d1e11825fe25c203b9be716fd91de"
server_key_iv: "e77ac0fe09392941d91755afaa1b6f40"
server_key_version: "encryptionkey@0"
I20260926 12:41:37.603754 26872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:37.620324 26872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:37.623831 26872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:37.624037 26872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:37.624650 26872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:37.625532 26872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:37.625605 26872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.625676 26872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:37.625726 26872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:37.635335 26872 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:38735
I20260926 12:41:37.635351 27000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:38735 every 8 connection(s)
I20260926 12:41:37.636363 26872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:41:37.643887 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 26872
I20260926 12:41:37.644038 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:41:37.644397 11138 external_mini_cluster.cc:1428] Setting key 8577343ba875c8760a11b1cd3cd7bbf4
Sep 26 12:41:37 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426497, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:37.652532 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.638458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:47771 (local address 127.10.224.190:34831)
0926 12:41:37.638632 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.638638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.639822 (+ 1184us) server_negotiation.cc:407] Connection header received
0926 12:41:37.641319 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.641322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.641375 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.641461 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:37.643008 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.643718 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.645533 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.645798 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.645927 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.648958 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:37.648961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:37.648964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:37.648989 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:37.650369 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:37.650898 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:37.650903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:37.650905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:37.650988 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:37.651356 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:37.651361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:37.651363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:37.651615 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:37.651731 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.652238 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.652365 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:41:37.653646 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.638976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:47771)
0926 12:41:37.639610 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.639685 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:37.641023 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:37.641659 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:37.641666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:37.642109 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:37.642830 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.642845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.643877 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.643881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.645387 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.645399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.645976 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.645980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.646199 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.646925 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:37.646943 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:37.648778 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:37.650490 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:37.650497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:37.650524 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:37.650793 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:37.651090 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:37.651093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:37.651095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:37.651259 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:37.651728 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:37.651736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:37.651994 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 12:41:37.652202 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 12:41:37.652545 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":127,"threads_started":1}
I20260926 12:41:37.655057 27001 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34831
I20260926 12:41:37.655365 27001 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:37.655993 27001 heartbeater.cc:511] Master 127.10.224.190:34831 requested a full tablet report, sending...
I20260926 12:41:37.657162 26542 ts_manager.cc:194] Registered new tserver with Master: 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735)
I20260926 12:41:37.657891 26542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:47771
I20260926 12:41:37.658591 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:41:37.668051 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.660730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.10.224.190:34831)
0926 12:41:37.660888 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.660892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.661032 (+ 140us) server_negotiation.cc:407] Connection header received
0926 12:41:37.661120 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.661123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.661165 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.661276 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:37.661989 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.662517 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.663367 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.663696 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.663795 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.665205 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:37.665208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:37.665210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:37.665233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:37.666510 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:37.666897 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:37.666900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:37.666901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:37.666972 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:37.667192 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:37.667194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:37.667195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:37.667439 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:37.667616 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.667818 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.667922 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 12:41:37.671027 26541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:41:37.674353 26541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:41:37.690024 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.684401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41695 (local address 127.0.0.1:60300)
0926 12:41:37.684995 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.685009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:37.685090 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:37.686103 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:37.686106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:37.686123 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:37.686207 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.686213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.687228 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.687233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.688438 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.688453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.689408 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.689412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.689529 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.689573 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 12:41:37.689694 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 12:41:37.689849 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":126,"thread_start_us":103,"threads_started":1}
I20260926 12:41:37.690035 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.684867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.10.224.130:41695)
0926 12:41:37.685692 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.685703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.685725 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:41:37.685802 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.685809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.685969 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.686114 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:37.686288 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.687078 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.688576 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.689277 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.689404 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.689544 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.689749 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.689856 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":104,"threads_started":1}
I20260926 12:41:37.690627 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.683415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38735 (local address 127.0.0.1:59794)
0926 12:41:37.684389 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.684446 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:37.684647 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:37.685420 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:37.685425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:37.685478 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:37.685632 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.685637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.687143 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.687146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.688895 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.688907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.690194 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.690199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.690293 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.690317 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:41:37.690390 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:41:37.690479 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":200,"spinlock_wait_cycles":5376,"thread_start_us":113,"threads_started":1}
I20260926 12:41:37.690815 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.683732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.10.224.129:39179)
0926 12:41:37.684146 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.684170 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.684687 (+ 517us) server_negotiation.cc:407] Connection header received
0926 12:41:37.684926 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.684932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.685139 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.685278 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:37.685810 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.686975 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.689050 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.689753 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.689861 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.689978 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.690558 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.690666 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":93,"threads_started":1}
I20260926 12:41:37.691150 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.683552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.10.224.131:38735)
0926 12:41:37.684440 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.684454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.684536 (+ 82us) server_negotiation.cc:407] Connection header received
0926 12:41:37.684843 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.684849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.685086 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.685226 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:37.685751 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.687003 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.689039 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.689882 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.690001 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.690122 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.690900 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.691005 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":164,"threads_started":1}
I20260926 12:41:37.690495 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.683491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:39179 (local address 127.0.0.1:34586)
0926 12:41:37.684581 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.684595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:37.684766 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:37.685449 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:37.685452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:37.685479 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:37.685689 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.685699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.687143 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.687146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.688899 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.688907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.690041 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.690047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.690166 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.690201 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 12:41:37.690272 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:41:37.690350 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":342,"spinlock_wait_cycles":5888,"thread_start_us":159,"threads_started":1}
I20260926 12:41:37.693611 26801 tablet_service.cc:1467] Processing CreateTablet for tablet ee5d9aac54c5402db36cd571866359f4 (DEFAULT_TABLE table=test-table [id=75e70b2bbb034dc3963bed88bbe6f3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:37.694293 26935 tablet_service.cc:1467] Processing CreateTablet for tablet ee5d9aac54c5402db36cd571866359f4 (DEFAULT_TABLE table=test-table [id=75e70b2bbb034dc3963bed88bbe6f3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:37.694291 26667 tablet_service.cc:1467] Processing CreateTablet for tablet ee5d9aac54c5402db36cd571866359f4 (DEFAULT_TABLE table=test-table [id=75e70b2bbb034dc3963bed88bbe6f3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:37.694753 26801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5d9aac54c5402db36cd571866359f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:37.695302 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5d9aac54c5402db36cd571866359f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:37.695309 26935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5d9aac54c5402db36cd571866359f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:37.702078 27018 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Bootstrap starting.
I20260926 12:41:37.703145 27020 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: Bootstrap starting.
I20260926 12:41:37.703857 27019 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Bootstrap starting.
I20260926 12:41:37.704990 27018 tablet_bootstrap.cc:654] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:37.705694 27020 tablet_bootstrap.cc:654] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:37.705925 27018 log.cc:824] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:37.705956 27019 tablet_bootstrap.cc:654] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:37.706578 27020 log.cc:824] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:37.706751 27019 log.cc:824] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:37.708277 27018 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: No bootstrap required, opened a new log
I20260926 12:41:37.708499 27018 ts_tablet_manager.cc:1397] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260926 12:41:37.708673 27020 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: No bootstrap required, opened a new log
I20260926 12:41:37.708719 27019 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: No bootstrap required, opened a new log
I20260926 12:41:37.708858 27019 ts_tablet_manager.cc:1397] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260926 12:41:37.708860 27020 ts_tablet_manager.cc:1397] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:41:37.711570 27018 raft_consensus.cc:348] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.711881 27018 raft_consensus.cc:374] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:37.711935 27018 raft_consensus.cc:729] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2654cb4c5fe44695bec5770bd3585be5, State: Initialized, Role: FOLLOWER
I20260926 12:41:37.712447 27018 consensus_queue.cc:260] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.712850 27020 raft_consensus.cc:348] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.713138 27020 raft_consensus.cc:374] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:37.713197 27020 raft_consensus.cc:729] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fe564b62434feead95d2ec809e166c, State: Initialized, Role: FOLLOWER
I20260926 12:41:37.713222 27019 raft_consensus.cc:348] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.713469 27019 raft_consensus.cc:374] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:37.713526 27019 raft_consensus.cc:729] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c7e1a2af6fa4922ae0586af5dd4ac2d, State: Initialized, Role: FOLLOWER
I20260926 12:41:37.713562 27018 ts_tablet_manager.cc:1428] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260926 12:41:37.713630 27020 consensus_queue.cc:260] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.713956 26867 heartbeater.cc:503] Master 127.10.224.190:34831 was elected leader, sending a full tablet report...
I20260926 12:41:37.713951 27019 consensus_queue.cc:260] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.714597 27020 ts_tablet_manager.cc:1428] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:41:37.714830 27019 ts_tablet_manager.cc:1428] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260926 12:41:37.714985 27001 heartbeater.cc:503] Master 127.10.224.190:34831 was elected leader, sending a full tablet report...
I20260926 12:41:37.715159 26733 heartbeater.cc:503] Master 127.10.224.190:34831 was elected leader, sending a full tablet report...
I20260926 12:41:37.728189 27026 raft_consensus.cc:482] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:37.728392 27026 raft_consensus.cc:504] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.729497 27026 leader_election.cc:288] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
I20260926 12:41:37.733909 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.729958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:50203 (local address 127.10.224.131:38735)
0926 12:41:37.730114 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.730119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.730332 (+ 213us) server_negotiation.cc:407] Connection header received
0926 12:41:37.730538 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.730545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.730606 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.730705 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:37.731102 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.731837 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.732913 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.733330 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.733479 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.733521 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.733702 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.733760 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:41:37.733928 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.729897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38735 (local address 127.10.224.129:50203)
0926 12:41:37.730233 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.730246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:37.730407 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:37.730762 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:37.730766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:37.730786 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:37.730930 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.730939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.731955 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.731960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.732780 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.732790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.733480 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.733483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.733578 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.733604 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:41:37.733681 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:41:37.733751 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":150,"threads_started":1}
I20260926 12:41:37.734537 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.730444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41695 (local address 127.10.224.129:34337)
0926 12:41:37.730786 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.730799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:37.730913 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:37.731132 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:37.731134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:37.731151 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:37.731260 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.731265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.732334 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.732338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.733077 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:37.733086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.733933 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.733939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:37.734273 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.734305 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 12:41:37.734378 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:41:37.734432 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":105,"threads_started":1}
I20260926 12:41:37.734642 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.730646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:34337 (local address 127.10.224.130:41695)
0926 12:41:37.730782 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.730787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.730882 (+ 95us) server_negotiation.cc:407] Connection header received
0926 12:41:37.731001 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.731004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.731053 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.731125 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:37.731658 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.732219 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.733225 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.733808 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.733920 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.733980 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.734432 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.734512 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:41:37.734586 26955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5d9aac54c5402db36cd571866359f4" candidate_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fe564b62434feead95d2ec809e166c" is_pre_election: true
I20260926 12:41:37.734913 26955 raft_consensus.cc:2454] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c7e1a2af6fa4922ae0586af5dd4ac2d in term 0.
I20260926 12:41:37.735298 26821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5d9aac54c5402db36cd571866359f4" candidate_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2654cb4c5fe44695bec5770bd3585be5" is_pre_election: true
I20260926 12:41:37.735507 26621 leader_election.cc:302] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fe564b62434feead95d2ec809e166c, 9c7e1a2af6fa4922ae0586af5dd4ac2d; no voters:
I20260926 12:41:37.735610 26821 raft_consensus.cc:2454] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c7e1a2af6fa4922ae0586af5dd4ac2d in term 0.
I20260926 12:41:37.735836 27026 raft_consensus.cc:2781] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:41:37.735939 27026 raft_consensus.cc:482] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:41:37.735999 27026 raft_consensus.cc:3037] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:37.737059 27026 raft_consensus.cc:504] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.737411 27026 leader_election.cc:288] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 1 election: Requested vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
I20260926 12:41:37.737792 26955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5d9aac54c5402db36cd571866359f4" candidate_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fe564b62434feead95d2ec809e166c"
I20260926 12:41:37.737838 26821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5d9aac54c5402db36cd571866359f4" candidate_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2654cb4c5fe44695bec5770bd3585be5"
I20260926 12:41:37.737911 26955 raft_consensus.cc:3037] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:37.737987 26821 raft_consensus.cc:3037] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:37.738999 26955 raft_consensus.cc:2454] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c7e1a2af6fa4922ae0586af5dd4ac2d in term 1.
I20260926 12:41:37.739135 26821 raft_consensus.cc:2454] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c7e1a2af6fa4922ae0586af5dd4ac2d in term 1.
I20260926 12:41:37.739378 26621 leader_election.cc:302] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fe564b62434feead95d2ec809e166c, 9c7e1a2af6fa4922ae0586af5dd4ac2d; no voters:
I20260926 12:41:37.739609 27026 raft_consensus.cc:2781] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:37.739842 27026 raft_consensus.cc:686] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 LEADER]: Becoming Leader. State: Replica: 9c7e1a2af6fa4922ae0586af5dd4ac2d, State: Running, Role: LEADER
I20260926 12:41:37.740185 27026 consensus_queue.cc:237] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:37.743782 26542 catalog_manager.cc:5495] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c7e1a2af6fa4922ae0586af5dd4ac2d (127.10.224.129). New cstate: current_term: 1 leader_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } health_report { overall_health: UNKNOWN } } }
I20260926 12:41:37.768862 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.765285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.10.224.129:39179)
0926 12:41:37.765430 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:37.765434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:37.765449 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:41:37.765530 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:37.765532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:37.765587 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:37.765680 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:41:37.765974 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.766486 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.767393 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:37.767621 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:37.767695 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:37.767957 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:41:37.768429 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:41:37.768520 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:37.768619 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 12:41:37.768673 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:41:37.775360 26821 raft_consensus.cc:1261] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 1 FOLLOWER]: Refusing update from remote peer 9c7e1a2af6fa4922ae0586af5dd4ac2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:37.775360 26955 raft_consensus.cc:1261] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 1 FOLLOWER]: Refusing update from remote peer 9c7e1a2af6fa4922ae0586af5dd4ac2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:37.776180 27026 consensus_queue.cc:1035] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:37.776453 27029 consensus_queue.cc:1035] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:37.795732 26540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:37.795914 26540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:37.798679 26540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Sending DeleteTablet for 3 replicas of tablet ee5d9aac54c5402db36cd571866359f4
I20260926 12:41:37.799644 26667 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5d9aac54c5402db36cd571866359f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:37 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:34586
I20260926 12:41:37.799722 26935 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5d9aac54c5402db36cd571866359f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:37 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:59794
I20260926 12:41:37.799820 26801 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5d9aac54c5402db36cd571866359f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:37 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:60300
I20260926 12:41:37.800341 27044 tablet_replica.cc:330] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: stopping tablet replica
I20260926 12:41:37.800508 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 26604
I20260926 12:41:37.800727 27044 raft_consensus.cc:2229] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:37.802194 27044 raft_consensus.cc:2258] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:37.802677 27043 tablet_replica.cc:330] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: stopping tablet replica
I20260926 12:41:37.803025 27043 raft_consensus.cc:2229] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:37.803339 27043 raft_consensus.cc:2258] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:37.803833 27044 ts_tablet_manager.cc:1905] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:37.805095 27043 ts_tablet_manager.cc:1905] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:37.806938 27044 ts_tablet_manager.cc:1918] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:37.807044 27044 log.cc:1196] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/ee5d9aac54c5402db36cd571866359f4
I20260926 12:41:37.807381 27044 ts_tablet_manager.cc:1939] T ee5d9aac54c5402db36cd571866359f4 P 17fe564b62434feead95d2ec809e166c: Deleting consensus metadata
W20260926 12:41:37.807667 26526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:39179 (error 108)
I20260926 12:41:37.808314 27043 ts_tablet_manager.cc:1918] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:37.808415 27043 log.cc:1196] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/ee5d9aac54c5402db36cd571866359f4
I20260926 12:41:37.808614 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 26738
I20260926 12:41:37.808739 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.808422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:39179 (local address 127.0.0.1:34606)
0926 12:41:37.808562 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.808654 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 12:41:37.809401 26526 catalog_manager.cc:4584] TS 9c7e1a2af6fa4922ae0586af5dd4ac2d (127.10.224.129:39179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee5d9aac54c5402db36cd571866359f4: Network error: Client connection negotiation failed: client connection to 127.10.224.129:39179: connect: Connection refused (error 111)
I20260926 12:41:37.812476 26527 catalog_manager.cc:4841] TS 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735): tablet ee5d9aac54c5402db36cd571866359f4 (table test-table [id=75e70b2bbb034dc3963bed88bbe6f3c1]) successfully deleted
I20260926 12:41:37.815001 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 26872
I20260926 12:41:37.815634 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:37.815190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41695 (local address 127.0.0.1:60316)
0926 12:41:37.815422 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:37.815491 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.130:41695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 12:41:37.821789 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 26510
I20260926 12:41:57.828770 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34831
--webserver_interface=127.10.224.190
--webserver_port=35669
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:34831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:41:57.941468 27046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:57.941766 27046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:57.941807 27046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:57.945832 27046 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:41:57.945891 27046 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:41:57.945907 27046 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:41:57.945922 27046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:41:57.945941 27046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:57.945959 27046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:57.945976 27046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:41:57.945991 27046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:41:57.951680 27046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:34831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=35669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:57.952797 27046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:57.953816 27046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:57.969112 27052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:57.969200 27055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:57.969336 27046 server_base.cc:1034] running on GCE node
W20260926 12:41:57.969431 27053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:57.970268 27046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:57.971580 27046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:57.972754 27046 hybrid_clock.cc:648] HybridClock initialized: now 1790426517972724 us; error 45 us; skew 500 ppm
Sep 26 12:41:57 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426517, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:57.975494 27046 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:41:57.976452 27046 webserver.cc:460] Webserver started at http://127.10.224.190:35669/ using document root <none> and password file <none>
I20260926 12:41:57.977066 27046 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:57.977210 27046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:57.981288 27046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 12:41:57.983650 27062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:57.984954 27046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:57.985110 27046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "34c53f617429429d8671bae159514186"
format_stamp: "Formatted at 2026-09-26 12:41:36 on dist-test-slave-m3pj"
server_key: "ef3adbed4ecb44695a7a9ec9ddce6e16"
server_key_iv: "13d13ce09218cdf4923f3aa251243719"
server_key_version: "encryptionkey@0"
I20260926 12:41:57.985744 27046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:58.008467 27046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:58.011960 27046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:58.012187 27046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:58.019780 27046 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:34831
I20260926 12:41:58.020126 27114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:34831 every 8 connection(s)
I20260926 12:41:58.020936 27046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:41:58.025661 27115 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:41:58.025768 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 27046
I20260926 12:41:58.026207 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:39179
--local_ip_for_outbound_sockets=127.10.224.129
--tserver_master_addrs=127.10.224.190:34831
--webserver_port=33163
--webserver_interface=127.10.224.129
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:58.027607 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Bootstrap starting.
I20260926 12:41:58.052023 27115 log.cc:824] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:58.062881 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:41:58.063308 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Bootstrap complete.
I20260926 12:41:58.068621 27115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:41:58.069167 27115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c53f617429429d8671bae159514186, State: Initialized, Role: FOLLOWER
I20260926 12:41:58.069761 27115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:41:58.069911 27115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:41:58.069984 27115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:41:58.070082 27115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:41:58.072535 27115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:41:58.072901 27115 leader_election.cc:302] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34c53f617429429d8671bae159514186; no voters:
I20260926 12:41:58.073261 27115 leader_election.cc:288] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:41:58.073508 27119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:41:58.074038 27119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 2 LEADER]: Becoming Leader. State: Replica: 34c53f617429429d8671bae159514186, State: Running, Role: LEADER
I20260926 12:41:58.074546 27115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:41:58.074723 27119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:41:58.076995 27120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34c53f617429429d8671bae159514186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } } }
I20260926 12:41:58.077471 27120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:58.077265 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34c53f617429429d8671bae159514186. Latest consensus state: current_term: 2 leader_uuid: "34c53f617429429d8671bae159514186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } } }
I20260926 12:41:58.077683 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: This master's current role is: LEADER
I20260926 12:41:58.081609 27133 catalog_manager.cc:1259] Loaded cluster ID: df17f2acd7834052a64935ddd6011598
I20260926 12:41:58.081684 27133 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: loading cluster ID for follower catalog manager: success
I20260926 12:41:58.084568 27133 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: acquiring CA information for follower catalog manager: success
I20260926 12:41:58.086351 27133 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:41:58.086597 27134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:41:58.089558 27134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:41:58.090166 27134 catalog_manager.cc:1259] Loaded cluster ID: df17f2acd7834052a64935ddd6011598
I20260926 12:41:58.090240 27134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:41:58.091058 27134 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:41:58.091622 27134 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Loaded TSK: 0
I20260926 12:41:58.105167 27134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Generated new TSK 1
I20260926 12:41:58.105441 27134 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:41:58.156672 27117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:58.156996 27117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:58.157081 27117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:58.161324 27117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:58.161413 27117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:58.161499 27117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:41:58.167268 27117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:39179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=33163
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:58.168375 27117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:58.169406 27117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:58.180187 27145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:58.180198 27142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:58.180227 27143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:58.181036 27117 server_base.cc:1034] running on GCE node
I20260926 12:41:58.181509 27117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:58.182318 27117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:58.183531 27117 hybrid_clock.cc:648] HybridClock initialized: now 1790426518183473 us; error 53 us; skew 500 ppm
Sep 26 12:41:58 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426518, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:58.187016 27117 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:41:58.188021 27117 webserver.cc:460] Webserver started at http://127.10.224.129:33163/ using document root <none> and password file <none>
I20260926 12:41:58.188629 27117 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:58.188735 27117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:58.193046 27117 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:41:58.195293 27152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:58.196689 27117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:58.196828 27117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "9422c84e0487a737454548502d20cadc"
server_key_iv: "573c40b711a7215663e5c7b7b3d740a3"
server_key_version: "encryptionkey@0"
I20260926 12:41:58.197393 27117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:58.230465 27117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:58.233987 27117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:58.234216 27117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:58.234835 27117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:58.236006 27159 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:41:58.240051 27117 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:41:58.240144 27117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 12:41:58.240234 27117 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:41:58.241978 27117 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:41:58.242044 27117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:41:58.242314 27159 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Bootstrap starting.
I20260926 12:41:58.252455 27117 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:39179
I20260926 12:41:58.252542 27266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:39179 every 8 connection(s)
I20260926 12:41:58.253427 27117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:41:58.254042 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 27117
I20260926 12:41:58.254619 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:41695
--local_ip_for_outbound_sockets=127.10.224.130
--tserver_master_addrs=127.10.224.190:34831
--webserver_port=39953
--webserver_interface=127.10.224.130
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:41:58.263263 27159 log.cc:824] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:41:58 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426518, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:58.271445 27159 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:41:58.271953 27159 tablet_bootstrap.cc:492] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Bootstrap complete.
I20260926 12:41:58.273396 27159 ts_tablet_manager.cc:1397] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.012s
I20260926 12:41:58.276957 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.255632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:54967 (local address 127.10.224.190:34831)
0926 12:41:58.255985 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.256006 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.257902 (+ 1896us) server_negotiation.cc:407] Connection header received
0926 12:41:58.259558 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.259583 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.260167 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.260600 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:58.262245 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.264359 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.266327 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.267033 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.267211 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.270632 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:58.270638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:58.270655 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:58.270698 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:58.273045 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.273816 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.273823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.273829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.273986 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.274383 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.274386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.274388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.274876 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:58.275052 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.275727 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.276040 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":119,"threads_started":1}
I20260926 12:41:58.276963 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.255647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:54967)
0926 12:41:58.257674 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.257733 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:58.258987 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:58.260680 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:58.260692 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:58.261249 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:58.262010 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.262037 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.264555 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.264562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.266170 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.266186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.267329 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.267334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.267574 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.268451 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:58.268472 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:58.270387 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:58.273184 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:58.273193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:58.273212 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:58.273655 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:58.274099 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:58.274103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:58.274105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:58.274279 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:58.275031 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:58.275038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:58.275326 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 12:41:58.275564 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 12:41:58.275964 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"thread_start_us":143,"threads_started":1}
I20260926 12:41:58.276744 27159 raft_consensus.cc:348] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:58.277315 27159 raft_consensus.cc:729] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c7e1a2af6fa4922ae0586af5dd4ac2d, State: Initialized, Role: FOLLOWER
I20260926 12:41:58.277825 27159 consensus_queue.cc:260] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } }
I20260926 12:41:58.278810 27159 ts_tablet_manager.cc:1428] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:41:58.280086 27267 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34831
I20260926 12:41:58.280400 27267 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:58.280892 27267 heartbeater.cc:511] Master 127.10.224.190:34831 requested a full tablet report, sending...
I20260926 12:41:58.283595 27079 ts_manager.cc:194] Registered new tserver with Master: 9c7e1a2af6fa4922ae0586af5dd4ac2d (127.10.224.129:39179)
I20260926 12:41:58.287741 27079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:54967
I20260926 12:41:58.290645 27267 heartbeater.cc:503] Master 127.10.224.190:34831 was elected leader, sending a full tablet report...
Sep 26 12:41:58 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426517, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.129@KRBTEST.COM
I20260926 12:41:58.298178 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.287258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.10.224.129:39179)
0926 12:41:58.288100 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.288109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.288127 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:41:58.288187 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.288191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.288353 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.288503 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:58.289199 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.290187 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.291212 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.291698 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.291822 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.294422 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:58.294431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:58.294449 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:58.294473 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:58.296207 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.296713 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.296718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.296719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.296797 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.297074 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.297077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.297079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.297335 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:58.297477 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.297816 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.298028 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":756,"thread_start_us":125,"threads_started":1}
I20260926 12:41:58.298178 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.287129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:39179 (local address 127.0.0.1:59544)
0926 12:41:58.287562 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.287607 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:58.287736 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:58.288508 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:58.288514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:58.288926 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:58.289043 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.289049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.290387 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.290391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.291116 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.291130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.291839 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.291843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.291955 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.292696 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:58.292711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:58.294273 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:58.296343 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:58.296352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:58.296358 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:58.296592 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:58.296891 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:58.296894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:58.296896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:58.296979 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:58.297450 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:58.297457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:58.297599 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 12:41:58.297745 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 12:41:58.297990 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":187,"threads_started":1}
I20260926 12:41:58.299211 27201 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5d9aac54c5402db36cd571866359f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:37 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:59544
I20260926 12:41:58.300192 27276 tablet_replica.cc:330] stopping tablet replica
I20260926 12:41:58.300529 27276 raft_consensus.cc:2229] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:58.300800 27276 raft_consensus.cc:2258] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:58.301935 27276 ts_tablet_manager.cc:1905] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:58.306587 27276 ts_tablet_manager.cc:1918] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:58.306691 27276 log.cc:1196] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal/wals/ee5d9aac54c5402db36cd571866359f4
I20260926 12:41:58.306953 27276 ts_tablet_manager.cc:1939] T ee5d9aac54c5402db36cd571866359f4 P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Deleting consensus metadata
I20260926 12:41:58.309120 27063 catalog_manager.cc:4841] TS 9c7e1a2af6fa4922ae0586af5dd4ac2d (127.10.224.129:39179): tablet ee5d9aac54c5402db36cd571866359f4 (table test-table [id=75e70b2bbb034dc3963bed88bbe6f3c1]) successfully deleted
W20260926 12:41:58.385181 27271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:58.385499 27271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:58.385545 27271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:58.389978 27271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:58.390058 27271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:58.390122 27271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:41:58.395916 27271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:41695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=39953
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:58.397038 27271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:58.398080 27271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:58.408074 27284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:58.408061 27281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:58.408066 27282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:58.408183 27271 server_base.cc:1034] running on GCE node
I20260926 12:41:58.408785 27271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:58.409401 27271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:58.410606 27271 hybrid_clock.cc:648] HybridClock initialized: now 1790426518410570 us; error 47 us; skew 500 ppm
Sep 26 12:41:58 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426518, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:58.413408 27271 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:41:58.414338 27271 webserver.cc:460] Webserver started at http://127.10.224.130:39953/ using document root <none> and password file <none>
I20260926 12:41:58.414911 27271 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:58.415015 27271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:58.418852 27271 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:41:58.421026 27291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:58.422273 27271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:41:58.422429 27271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "2654cb4c5fe44695bec5770bd3585be5"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "3205e7694c835876a041fc4b14f3806a"
server_key_iv: "049116ad16398af7388f29efce9e2e08"
server_key_version: "encryptionkey@0"
I20260926 12:41:58.422977 27271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:58.444774 27271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:58.447765 27271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:58.447985 27271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:58.448589 27271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:58.449690 27298 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 12:41:58.453775 27298 ts_tablet_manager.cc:1852] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 12:41:58.454159 27298 ts_tablet_manager.cc:1905] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:58.458187 27298 ts_tablet_manager.cc:1918] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:58.458448 27298 log.cc:1196] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/ee5d9aac54c5402db36cd571866359f4
I20260926 12:41:58.458868 27298 ts_tablet_manager.cc:1939] T ee5d9aac54c5402db36cd571866359f4 P 2654cb4c5fe44695bec5770bd3585be5: Deleting consensus metadata
I20260926 12:41:58.459198 27271 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 12:41:58.459270 27271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260926 12:41:58.459379 27271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:58.459589 27271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:58.468770 27271 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:41695
I20260926 12:41:58.468806 27405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:41695 every 8 connection(s)
I20260926 12:41:58.469650 27271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:41:58.472688 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 27271
I20260926 12:41:58.473194 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:38735
--local_ip_for_outbound_sockets=127.10.224.131
--tserver_master_addrs=127.10.224.190:34831
--webserver_port=35681
--webserver_interface=127.10.224.131
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:41:58 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426518, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:58.483698 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.471382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:45449 (local address 127.10.224.190:34831)
0926 12:41:58.471555 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.471560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.472472 (+ 912us) server_negotiation.cc:407] Connection header received
0926 12:41:58.473762 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.473765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.473816 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.473890 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:58.475331 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.475968 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.477160 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.477403 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.477516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.480080 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:58.480083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:58.480085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:58.480110 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:58.481630 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.482146 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.482151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.482152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.482244 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.482589 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.482594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.482596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.482777 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:58.482883 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.483369 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.483508 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:41:58.484542 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.471837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:45449)
0926 12:41:58.472279 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.472348 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:58.473536 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:58.474092 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:58.474103 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:58.474642 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:58.475183 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.475197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.476100 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.476103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.476971 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.476980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.477553 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.477558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.477724 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.478453 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:58.478469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:58.479906 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:58.481750 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:58.481756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:58.481771 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:58.482041 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:58.482341 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:58.482344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:58.482346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:58.482491 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:58.482879 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:58.482887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:58.483134 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 12:41:58.483336 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 12:41:58.483694 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260926 12:41:58.485785 27406 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34831
I20260926 12:41:58.486052 27406 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:58.486562 27406 heartbeater.cc:511] Master 127.10.224.190:34831 requested a full tablet report, sending...
I20260926 12:41:58.487706 27079 ts_manager.cc:194] Registered new tserver with Master: 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
I20260926 12:41:58.488535 27079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:45449
W20260926 12:41:58.592095 27409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:41:58.592401 27409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:41:58.592451 27409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:41:58.596834 27409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:41:58.596915 27409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:41:58.596979 27409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:41:58.602774 27409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:38735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=35681
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:41:58.603992 27409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:41:58.605144 27409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:41:58.615072 27414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:58.615152 27409 server_base.cc:1034] running on GCE node
W20260926 12:41:58.615073 27417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:41:58.615077 27415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:41:58.615849 27409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:41:58.616484 27409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:41:58.617660 27409 hybrid_clock.cc:648] HybridClock initialized: now 1790426518617639 us; error 34 us; skew 500 ppm
Sep 26 12:41:58 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426518, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:41:58.620451 27409 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:41:58.621361 27409 webserver.cc:460] Webserver started at http://127.10.224.131:35681/ using document root <none> and password file <none>
I20260926 12:41:58.621938 27409 fs_manager.cc:359] Metadata directory not provided
I20260926 12:41:58.622046 27409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:41:58.626300 27409 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:41:58.628669 27424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:58.630136 27409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:41:58.630321 27409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "17fe564b62434feead95d2ec809e166c"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "af5d1e11825fe25c203b9be716fd91de"
server_key_iv: "e77ac0fe09392941d91755afaa1b6f40"
server_key_version: "encryptionkey@0"
I20260926 12:41:58.630945 27409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:41:58.659368 27409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:41:58.663008 27409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:41:58.663228 27409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:41:58.663913 27409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:41:58.664757 27409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:41:58.664805 27409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:58.664877 27409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:41:58.664911 27409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:41:58.674424 27409 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:38735
I20260926 12:41:58.674440 27537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:38735 every 8 connection(s)
I20260926 12:41:58.675650 27409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:41:58.680444 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 27409
Sep 26 12:41:58 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426518, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:41:58.691038 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.681272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.10.224.190:34831)
0926 12:41:58.681535 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.681539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.681633 (+ 94us) server_negotiation.cc:407] Connection header received
0926 12:41:58.681728 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.681731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.681772 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.681889 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:58.682674 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.683384 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.684795 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.685071 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.685169 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.686973 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:58.686976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:58.686978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:58.687001 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:58.688893 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.689488 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.689494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.689498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.689638 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.689955 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.689958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.689960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.690237 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:58.690479 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.690684 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.690810 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260926 12:41:58.692691 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.677655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:53525 (local address 127.10.224.190:34831)
0926 12:41:58.677809 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.677813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.678758 (+ 945us) server_negotiation.cc:407] Connection header received
0926 12:41:58.680737 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.680742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.680803 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.680910 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:58.682413 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.683168 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.685009 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.685305 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.685394 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.688407 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:58.688413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:58.688417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:58.688457 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:58.689985 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.690685 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.690688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.690690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.690780 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.691398 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.691402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.691404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.691666 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:58.691788 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.692409 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.692533 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:41:58.693604 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.678119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:53525)
0926 12:41:58.678597 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.678642 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:58.680467 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:58.681107 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:58.681114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:58.681646 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:41:58.682277 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.682292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.683306 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.683310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.684908 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.684919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.685490 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.685498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.685734 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.686277 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:41:58.686305 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:41:58.688198 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:41:58.690101 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:58.690107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:58.690127 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:58.690549 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:58.691054 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:41:58.691060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:41:58.691064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:41:58.691284 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:41:58.691797 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:41:58.691807 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:41:58.692147 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 12:41:58.692350 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 12:41:58.692734 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260926 12:41:58.694651 27538 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34831
I20260926 12:41:58.694929 27538 heartbeater.cc:464] Registering TS with master...
I20260926 12:41:58.695454 27538 heartbeater.cc:511] Master 127.10.224.190:34831 requested a full tablet report, sending...
I20260926 12:41:58.696664 27079 ts_manager.cc:194] Registered new tserver with Master: 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735)
I20260926 12:41:58.697222 27079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:53525
I20260926 12:41:58.704751 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:41:58.708927 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.705420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.10.224.190:34831)
0926 12:41:58.705570 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.705575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.705750 (+ 175us) server_negotiation.cc:407] Connection header received
0926 12:41:58.705897 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.705899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.706017 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.706125 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:41:58.706380 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.706912 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.707858 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.708069 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.708151 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.708410 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:41:58.708555 (+ 145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 12:41:58.708772 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36264: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 12:41:58.709023 27269 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36264: token expired
W20260926 12:41:58.709326 27006 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.224.190:34831)
I20260926 12:41:58.716660 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.709686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.10.224.190:34831)
0926 12:41:58.709804 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.709807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.709824 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:41:58.709939 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.709941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.709979 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.710048 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:41:58.710628 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.711092 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.711879 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.712105 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.712196 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.713514 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:41:58.713517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:41:58.713519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:41:58.713542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:41:58.715143 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.715512 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.715514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.715597 (+ 83us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.715671 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:41:58.715908 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:41:58.715910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:41:58.715911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:41:58.716081 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:41:58.716179 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.716365 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.716460 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:41:58.719110 27079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:41:58.722478 27079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:41:58.730768 27201 tablet_service.cc:1467] Processing CreateTablet for tablet 7165f274c2f64134bb3f72920a1b7b7d (DEFAULT_TABLE table=test-table [id=915fa166c0ee4bf086a6caafca0f923b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:58.731493 27201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7165f274c2f64134bb3f72920a1b7b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:58.734931 27159 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Bootstrap starting.
I20260926 12:41:58.735948 27159 tablet_bootstrap.cc:654] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:58.736426 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.729672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41695 (local address 127.0.0.1:56714)
0926 12:41:58.730542 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.730553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:58.730669 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:58.731303 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:58.731305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:58.731320 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:58.731385 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.731389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.732527 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.732530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.733736 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.733748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.735629 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.735634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.735757 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.735980 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 12:41:58.736133 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 12:41:58.736243 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":72,"thread_start_us":153,"threads_started":1}
I20260926 12:41:58.736985 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.729634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.10.224.131:38735)
0926 12:41:58.730207 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.730218 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.730239 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:41:58.730349 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.730355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.730595 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.730774 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:58.730966 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.732176 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.735345 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.736350 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.736499 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.736652 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.736726 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.736831 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":196,"threads_started":1}
I20260926 12:41:58.737035 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.729900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.10.224.130:41695)
0926 12:41:58.730418 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.730427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.730642 (+ 215us) server_negotiation.cc:407] Connection header received
0926 12:41:58.730824 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.730829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.731030 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.731165 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:58.731498 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.732397 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.733886 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.734913 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.735065 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.735253 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.736744 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.736860 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":122,"threads_started":1}
I20260926 12:41:58.736922 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.729419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38735 (local address 127.0.0.1:35748)
0926 12:41:58.729548 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.729565 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:58.729656 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:58.730749 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:58.730755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:58.730773 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:58.730883 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.730889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.732308 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.732311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.733736 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.733748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.736473 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.736478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.736597 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.736621 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:41:58.736694 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:41:58.736793 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":8}
I20260926 12:41:58.737618 27159 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: No bootstrap required, opened a new log
I20260926 12:41:58.737715 27159 ts_tablet_manager.cc:1397] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:41:58.738180 27159 raft_consensus.cc:348] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:41:58.738287 27159 raft_consensus.cc:374] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:58.738315 27159 raft_consensus.cc:729] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c7e1a2af6fa4922ae0586af5dd4ac2d, State: Initialized, Role: FOLLOWER
I20260926 12:41:58.738459 27159 consensus_queue.cc:260] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:41:58.738736 27159 ts_tablet_manager.cc:1428] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:41:58.738832 27340 tablet_service.cc:1467] Processing CreateTablet for tablet 7165f274c2f64134bb3f72920a1b7b7d (DEFAULT_TABLE table=test-table [id=915fa166c0ee4bf086a6caafca0f923b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:58.740108 27340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7165f274c2f64134bb3f72920a1b7b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:58.741063 27472 tablet_service.cc:1467] Processing CreateTablet for tablet 7165f274c2f64134bb3f72920a1b7b7d (DEFAULT_TABLE table=test-table [id=915fa166c0ee4bf086a6caafca0f923b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:41:58.742084 27472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7165f274c2f64134bb3f72920a1b7b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:41:58.745218 27298 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: Bootstrap starting.
I20260926 12:41:58.747339 27298 tablet_bootstrap.cc:654] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:58.748291 27298 log.cc:824] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:58.749727 27552 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Bootstrap starting.
I20260926 12:41:58.750530 27298 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: No bootstrap required, opened a new log
I20260926 12:41:58.750732 27298 ts_tablet_manager.cc:1397] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:41:58.752398 27552 tablet_bootstrap.cc:654] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Neither blocks nor log segments found. Creating new log.
I20260926 12:41:58.753180 27552 log.cc:824] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:41:58.753932 27298 raft_consensus.cc:348] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:41:58.754201 27298 raft_consensus.cc:374] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:58.754282 27298 raft_consensus.cc:729] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2654cb4c5fe44695bec5770bd3585be5, State: Initialized, Role: FOLLOWER
I20260926 12:41:58.754787 27298 consensus_queue.cc:260] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
W20260926 12:41:58.754868 27268 tablet.cc:2369] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:41:58.755355 27552 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: No bootstrap required, opened a new log
I20260926 12:41:58.755622 27552 ts_tablet_manager.cc:1397] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:41:58.755861 27298 ts_tablet_manager.cc:1428] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:41:58.756242 27406 heartbeater.cc:503] Master 127.10.224.190:34831 was elected leader, sending a full tablet report...
I20260926 12:41:58.759109 27552 raft_consensus.cc:348] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:41:58.759353 27552 raft_consensus.cc:374] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:41:58.759395 27552 raft_consensus.cc:729] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fe564b62434feead95d2ec809e166c, State: Initialized, Role: FOLLOWER
I20260926 12:41:58.759896 27552 consensus_queue.cc:260] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:41:58.760607 27538 heartbeater.cc:503] Master 127.10.224.190:34831 was elected leader, sending a full tablet report...
I20260926 12:41:58.760854 27552 ts_tablet_manager.cc:1428] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:41:58.895211 27554 raft_consensus.cc:482] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:41:58.895459 27554 raft_consensus.cc:504] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:41:58.896679 27554 leader_election.cc:288] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 9c7e1a2af6fa4922ae0586af5dd4ac2d (127.10.224.129:39179)
I20260926 12:41:58.901640 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.897048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:53019 (local address 127.10.224.129:39179)
0926 12:41:58.897337 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.897342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.897494 (+ 152us) server_negotiation.cc:407] Connection header received
0926 12:41:58.897552 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.897557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.897624 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.897728 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:58.898014 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.898740 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.900334 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.900989 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.901095 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.901241 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.901424 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.901503 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":114,"threads_started":1}
I20260926 12:41:58.901638 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.897203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:46919 (local address 127.10.224.131:38735)
0926 12:41:58.897337 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.897342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.897359 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:41:58.897420 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.897422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.897496 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.897584 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:41:58.897869 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.898674 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.900334 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.900934 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.901082 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.901141 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.901442 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.901509 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:41:58.901692 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.896960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38735 (local address 127.10.224.130:46919)
0926 12:41:58.897077 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.897094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:58.897196 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:58.897613 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:58.897615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:58.897629 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:58.897753 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.897760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.898786 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.898788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.900179 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.900186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.901063 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.901069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.901202 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.901275 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 12:41:58.901412 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 12:41:58.901464 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":73}
I20260926 12:41:58.901741 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.896963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:39179 (local address 127.10.224.130:53019)
0926 12:41:58.897394 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.897403 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 12:41:58.897487 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:41:58.897761 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:41:58.897764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:41:58.897778 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:41:58.897912 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.897917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.898865 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.898868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.900175 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:41:58.900186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.901098 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.901101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:41:58.901206 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.901279 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 12:41:58.901354 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 12:41:58.901446 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":127,"thread_start_us":141,"threads_started":1}
I20260926 12:41:58.902379 27492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7165f274c2f64134bb3f72920a1b7b7d" candidate_uuid: "2654cb4c5fe44695bec5770bd3585be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fe564b62434feead95d2ec809e166c" is_pre_election: true
I20260926 12:41:58.902376 27221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7165f274c2f64134bb3f72920a1b7b7d" candidate_uuid: "2654cb4c5fe44695bec5770bd3585be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" is_pre_election: true
I20260926 12:41:58.902694 27492 raft_consensus.cc:2454] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2654cb4c5fe44695bec5770bd3585be5 in term 0.
I20260926 12:41:58.902694 27221 raft_consensus.cc:2454] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2654cb4c5fe44695bec5770bd3585be5 in term 0.
I20260926 12:41:58.903242 27292 leader_election.cc:302] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17fe564b62434feead95d2ec809e166c, 2654cb4c5fe44695bec5770bd3585be5, 9c7e1a2af6fa4922ae0586af5dd4ac2d; no voters:
I20260926 12:41:58.903564 27554 raft_consensus.cc:2781] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:41:58.903649 27554 raft_consensus.cc:482] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:41:58.903702 27554 raft_consensus.cc:3037] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:58.904783 27554 raft_consensus.cc:504] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:41:58.905143 27554 leader_election.cc:288] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [CANDIDATE]: Term 1 election: Requested vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 9c7e1a2af6fa4922ae0586af5dd4ac2d (127.10.224.129:39179)
I20260926 12:41:58.905488 27221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7165f274c2f64134bb3f72920a1b7b7d" candidate_uuid: "2654cb4c5fe44695bec5770bd3585be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d"
I20260926 12:41:58.905493 27492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7165f274c2f64134bb3f72920a1b7b7d" candidate_uuid: "2654cb4c5fe44695bec5770bd3585be5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fe564b62434feead95d2ec809e166c"
I20260926 12:41:58.905624 27221 raft_consensus.cc:3037] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:58.905634 27492 raft_consensus.cc:3037] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:41:58.906672 27492 raft_consensus.cc:2454] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2654cb4c5fe44695bec5770bd3585be5 in term 1.
I20260926 12:41:58.906704 27221 raft_consensus.cc:2454] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2654cb4c5fe44695bec5770bd3585be5 in term 1.
I20260926 12:41:58.907006 27293 leader_election.cc:302] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fe564b62434feead95d2ec809e166c, 2654cb4c5fe44695bec5770bd3585be5; no voters:
I20260926 12:41:58.907178 27554 raft_consensus.cc:2781] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:41:58.907368 27554 raft_consensus.cc:686] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 1 LEADER]: Becoming Leader. State: Replica: 2654cb4c5fe44695bec5770bd3585be5, State: Running, Role: LEADER
I20260926 12:41:58.907786 27554 consensus_queue.cc:237] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:41:58.910943 27079 catalog_manager.cc:5495] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130). New cstate: current_term: 1 leader_uuid: "2654cb4c5fe44695bec5770bd3585be5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } health_report { overall_health: UNKNOWN } } }
W20260926 12:41:58.926666 27539 tablet.cc:2369] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:41:58.929479 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.925620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.10.224.130:41695)
0926 12:41:58.925762 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:41:58.925766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:41:58.925782 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:41:58.925907 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:41:58.925909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:41:58.925967 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:41:58.926075 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:41:58.926331 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.927042 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.928018 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:41:58.928230 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:41:58.928312 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:41:58.928643 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:41:58.929066 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:41:58.929156 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:41:58.929289 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 12:41:58.929347 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:41:58.935822 27492 raft_consensus.cc:1261] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 1 FOLLOWER]: Refusing update from remote peer 2654cb4c5fe44695bec5770bd3585be5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:58.935817 27221 raft_consensus.cc:1261] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Refusing update from remote peer 2654cb4c5fe44695bec5770bd3585be5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:41:58.936493 27554 consensus_queue.cc:1035] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:58.936718 27558 consensus_queue.cc:1035] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:41:58.944855 27562 mvcc.cc:205] Tried to move back new op lower bound from 7333587021552021504 to 7333587021448773632. Current Snapshot: MvccSnapshot[applied={T|T < 7333587021552021504}]
I20260926 12:41:58.952399 27079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:58.952586 27079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:41:58.954895 27079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Sending DeleteTablet for 3 replicas of tablet 7165f274c2f64134bb3f72920a1b7b7d
I20260926 12:41:58.955665 27472 tablet_service.cc:1514] Processing DeleteTablet for tablet 7165f274c2f64134bb3f72920a1b7b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:58 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:35748
I20260926 12:41:58.955677 27340 tablet_service.cc:1514] Processing DeleteTablet for tablet 7165f274c2f64134bb3f72920a1b7b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:58 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:56714
I20260926 12:41:58.955766 27201 tablet_service.cc:1514] Processing DeleteTablet for tablet 7165f274c2f64134bb3f72920a1b7b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:41:58 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:59544
I20260926 12:41:58.956012 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 27117
I20260926 12:41:58.956183 27572 tablet_replica.cc:330] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: stopping tablet replica
I20260926 12:41:58.956333 27572 raft_consensus.cc:2229] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:58.956660 27573 tablet_replica.cc:330] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: stopping tablet replica
I20260926 12:41:58.957015 27573 raft_consensus.cc:2229] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:41:58.957453 27573 raft_consensus.cc:2258] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:58.957862 27574 tablet_replica.cc:330] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: stopping tablet replica
I20260926 12:41:58.958163 27574 raft_consensus.cc:2229] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:41:58.958477 27574 raft_consensus.cc:2258] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:41:58.959473 27573 ts_tablet_manager.cc:1905] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:58.959848 27574 ts_tablet_manager.cc:1905] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:41:58.962680 27574 ts_tablet_manager.cc:1918] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:58.962787 27574 log.cc:1196] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/7165f274c2f64134bb3f72920a1b7b7d
I20260926 12:41:58.963148 27574 ts_tablet_manager.cc:1939] T 7165f274c2f64134bb3f72920a1b7b7d P 17fe564b62434feead95d2ec809e166c: Deleting consensus metadata
I20260926 12:41:58.963387 27573 ts_tablet_manager.cc:1918] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:41:58.963552 27573 log.cc:1196] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/7165f274c2f64134bb3f72920a1b7b7d
I20260926 12:41:58.963981 27573 ts_tablet_manager.cc:1939] T 7165f274c2f64134bb3f72920a1b7b7d P 2654cb4c5fe44695bec5770bd3585be5: Deleting consensus metadata
W20260926 12:41:58.964079 27063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:39179 (error 108)
I20260926 12:41:58.964563 27064 catalog_manager.cc:4841] TS 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735): tablet 7165f274c2f64134bb3f72920a1b7b7d (table test-table [id=915fa166c0ee4bf086a6caafca0f923b]) successfully deleted
I20260926 12:41:58.965310 27064 catalog_manager.cc:4841] TS 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695): tablet 7165f274c2f64134bb3f72920a1b7b7d (table test-table [id=915fa166c0ee4bf086a6caafca0f923b]) successfully deleted
I20260926 12:41:58.965502 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 27271
I20260926 12:41:58.965586 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:41:58.965256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:39179 (local address 127.0.0.1:59548)
0926 12:41:58.965403 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 12:41:58.965496 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 12:41:58.967024 27063 catalog_manager.cc:4584] TS 9c7e1a2af6fa4922ae0586af5dd4ac2d (127.10.224.129:39179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7165f274c2f64134bb3f72920a1b7b7d: Network error: Client connection negotiation failed: client connection to 127.10.224.129:39179: connect: Connection refused (error 111)
I20260926 12:41:58.972074 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 27409
I20260926 12:41:58.977824 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 27046
I20260926 12:42:18.984931 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34831
--webserver_interface=127.10.224.190
--webserver_port=35669
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:34831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:19.098851 27576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:19.099174 27576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:19.099216 27576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:19.103366 27576 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:42:19.103438 27576 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:42:19.103454 27576 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:42:19.103467 27576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:19.103487 27576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:19.103561 27576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:19.103595 27576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:19.103624 27576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:19.109153 27576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:34831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=35669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:19.110461 27576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:19.111567 27576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:42:19.120784 27576 server_base.cc:1034] running on GCE node
W20260926 12:42:19.120720 27582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:19.120729 27581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:19.120733 27584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:19.121587 27576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:19.122778 27576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:19.123943 27576 hybrid_clock.cc:648] HybridClock initialized: now 1790426539123911 us; error 48 us; skew 500 ppm
Sep 26 12:42:19 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426539, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:19.126847 27576 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:42:19.127871 27576 webserver.cc:460] Webserver started at http://127.10.224.190:35669/ using document root <none> and password file <none>
I20260926 12:42:19.128456 27576 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:19.128518 27576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:19.132561 27576 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:42:19.134843 27591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:19.136173 27576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:42:19.136317 27576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "34c53f617429429d8671bae159514186"
format_stamp: "Formatted at 2026-09-26 12:41:36 on dist-test-slave-m3pj"
server_key: "ef3adbed4ecb44695a7a9ec9ddce6e16"
server_key_iv: "13d13ce09218cdf4923f3aa251243719"
server_key_version: "encryptionkey@0"
I20260926 12:42:19.136893 27576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:19.156615 27576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:19.160403 27576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:19.160645 27576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:19.168767 27576 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:34831
I20260926 12:42:19.169097 27643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:34831 every 8 connection(s)
I20260926 12:42:19.170341 27576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:19.172227 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 27576
I20260926 12:42:19.172765 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:39179
--local_ip_for_outbound_sockets=127.10.224.129
--tserver_master_addrs=127.10.224.190:34831
--webserver_port=33163
--webserver_interface=127.10.224.129
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:42:19.175480 27644 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:42:19.178169 27644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Bootstrap starting.
I20260926 12:42:19.202603 27644 log.cc:824] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:19.216879 27644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:42:19.217422 27644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Bootstrap complete.
I20260926 12:42:19.223109 27644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:42:19.223839 27644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c53f617429429d8671bae159514186, State: Initialized, Role: FOLLOWER
I20260926 12:42:19.224380 27644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:42:19.224527 27644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:19.224623 27644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:19.224751 27644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:42:19.226778 27644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:42:19.227125 27644 leader_election.cc:302] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34c53f617429429d8671bae159514186; no voters:
I20260926 12:42:19.227454 27644 leader_election.cc:288] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 12:42:19.227622 27648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:42:19.227895 27648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [term 3 LEADER]: Becoming Leader. State: Replica: 34c53f617429429d8671bae159514186, State: Running, Role: LEADER
I20260926 12:42:19.228259 27648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } }
I20260926 12:42:19.228804 27644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:19.229707 27649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "34c53f617429429d8671bae159514186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } } }
I20260926 12:42:19.229859 27649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:19.230257 27650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34c53f617429429d8671bae159514186. Latest consensus state: current_term: 3 leader_uuid: "34c53f617429429d8671bae159514186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c53f617429429d8671bae159514186" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34831 } } }
I20260926 12:42:19.230398 27650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:19.230738 27657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:19.236383 27657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:19.237002 27657 catalog_manager.cc:1259] Loaded cluster ID: df17f2acd7834052a64935ddd6011598
I20260926 12:42:19.237061 27657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:19.239509 27657 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:19.240266 27657 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Loaded TSK: 0
I20260926 12:42:19.240445 27657 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34c53f617429429d8671bae159514186: Loaded TSK: 1
F20260926 12:42:19.241487 27657 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fccd08119ad google::LogMessage::Fail() at ??:0
@ 0x7fccd081488a google::LogMessage::SendToLog() at ??:0
@ 0x7fccd08114de google::LogMessage::Flush() at ??:0
@ 0x7fccd0812a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fccd34d518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fccd34cdf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fccdb1767bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fccdb15099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fccdb151178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fccdb192df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fccdb1b182b std::function<>::operator()() at ??:0
@ 0x7fccdb150b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fccdb151f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fccdb14e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fccdb192640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fccd2a633d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fccd2a63c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fccd2a653f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fccd2a54512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fccd40986db start_thread at ??:0
@ 0x7fcccf30f61f clone at ??:0
W20260926 12:42:19.334638 27646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:19.334910 27646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:19.334973 27646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:19.340350 27646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:19.340415 27646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:19.340534 27646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:42:19.347325 27646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:39179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=33163
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:19.348749 27646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:19.349974 27646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:19.361990 27691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:19.362020 27692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:19.362273 27646 server_base.cc:1034] running on GCE node
W20260926 12:42:19.362375 27694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:19.362902 27646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:19.363819 27646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:19.365041 27646 hybrid_clock.cc:648] HybridClock initialized: now 1790426539365015 us; error 39 us; skew 500 ppm
Sep 26 12:42:19 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426539, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:19.368687 27646 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:42:19.369920 27646 webserver.cc:460] Webserver started at http://127.10.224.129:33163/ using document root <none> and password file <none>
I20260926 12:42:19.370680 27646 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:19.370775 27646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:19.376603 27646 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:42:19.379665 27701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:19.381351 27646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:42:19.381492 27646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "9422c84e0487a737454548502d20cadc"
server_key_iv: "573c40b711a7215663e5c7b7b3d740a3"
server_key_version: "encryptionkey@0"
I20260926 12:42:19.382299 27646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:19.402876 27646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:19.406437 27646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:19.406661 27646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:19.407292 27646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:19.408519 27708 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:42:19.414283 27646 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:42:19.414439 27646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 12:42:19.414546 27646 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:42:19.416380 27646 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:42:19.416455 27646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 12:42:19.416762 27708 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Bootstrap starting.
I20260926 12:42:19.427623 27646 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:39179
I20260926 12:42:19.427803 27815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:39179 every 8 connection(s)
I20260926 12:42:19.428861 27646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:42:19.431180 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 27646
I20260926 12:42:19.431783 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:41695
--local_ip_for_outbound_sockets=127.10.224.130
--tserver_master_addrs=127.10.224.190:34831
--webserver_port=39953
--webserver_interface=127.10.224.130
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:42:19.440485 27708 log.cc:824] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:19.447893 27708 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:42:19.448299 27708 tablet_bootstrap.cc:492] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Bootstrap complete.
I20260926 12:42:19.449478 27708 ts_tablet_manager.cc:1397] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260926 12:42:19.452306 27708 raft_consensus.cc:348] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:42:19.452836 27708 raft_consensus.cc:729] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c7e1a2af6fa4922ae0586af5dd4ac2d, State: Initialized, Role: FOLLOWER
I20260926 12:42:19.453372 27708 consensus_queue.cc:260] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:42:19.454375 27708 ts_tablet_manager.cc:1428] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260926 12:42:19.522259 27818 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:42:19.430996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:53639)
0926 12:42:19.433584 (+ 2588us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.433655 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:19.434838 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:19.521412 (+ 86574us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2219,"thread_start_us":165,"threads_started":1}
W20260926 12:42:19.522809 27816 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:34831 (0 consecutive failures): Network error: Failed to ping master at 127.10.224.190:34831: Client connection negotiation failed: client connection to 127.10.224.190:34831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:42:19.523685 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:19.523343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:54735)
0926 12:42:19.523497 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.523624 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 12:42:19.524434 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:19.524203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:48989)
0926 12:42:19.524321 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.524376 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:42:19.524942 27816 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:42:19.573622 27819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:19.573945 27819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:19.574023 27819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:19.578227 27819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:19.578361 27819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:19.578464 27819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:42:19.584080 27819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:41695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=39953
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:19.585304 27819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:19.586431 27819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:19.597096 27827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:19.597076 27826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:19.597076 27829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:19.597216 27819 server_base.cc:1034] running on GCE node
I20260926 12:42:19.597793 27819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:19.598476 27819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:19.599665 27819 hybrid_clock.cc:648] HybridClock initialized: now 1790426539599623 us; error 53 us; skew 500 ppm
Sep 26 12:42:19 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426539, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:19.602399 27819 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:42:19.603286 27819 webserver.cc:460] Webserver started at http://127.10.224.130:39953/ using document root <none> and password file <none>
I20260926 12:42:19.603922 27819 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:19.604028 27819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:19.608047 27819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:42:19.610235 27836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:19.611670 27819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:19.611801 27819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "2654cb4c5fe44695bec5770bd3585be5"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "3205e7694c835876a041fc4b14f3806a"
server_key_iv: "049116ad16398af7388f29efce9e2e08"
server_key_version: "encryptionkey@0"
I20260926 12:42:19.612378 27819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:19.640141 27819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:19.643453 27819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:19.643702 27819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:19.644322 27819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:19.645179 27819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:19.645249 27819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:19.645321 27819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:19.645373 27819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:19.655629 27819 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:41695
I20260926 12:42:19.655646 27949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:41695 every 8 connection(s)
I20260926 12:42:19.656723 27819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:42:19.660804 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:19.658955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:41947)
0926 12:42:19.659512 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.660010 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":133,"threads_started":1}
I20260926 12:42:19.661118 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 27819
W20260926 12:42:19.661526 27950 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:34831 (0 consecutive failures): Network error: Failed to ping master at 127.10.224.190:34831: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
I20260926 12:42:19.661541 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:38735
--local_ip_for_outbound_sockets=127.10.224.131
--tserver_master_addrs=127.10.224.190:34831
--webserver_port=35681
--webserver_interface=127.10.224.131
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:42:19.662184 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:19.661960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:34595)
0926 12:42:19.662079 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.662133 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 12:42:19.662783 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:19.662579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:55701)
0926 12:42:19.662681 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.662727 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 12:42:19.662962 27950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:42:19.775216 27953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:19.775565 27953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:19.775614 27953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:19.779886 27953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:19.779964 27953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:19.780030 27953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:42:19.785660 27953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:38735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=35681
--tserver_master_addrs=127.10.224.190:34831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:19.786806 27953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:19.787973 27953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:19.798463 27958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:19.798463 27959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:19.798534 27961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:19.798782 27953 server_base.cc:1034] running on GCE node
I20260926 12:42:19.799216 27953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:19.800015 27953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:19.801208 27953 hybrid_clock.cc:648] HybridClock initialized: now 1790426539801179 us; error 46 us; skew 500 ppm
Sep 26 12:42:19 dist-test-slave-m3pj krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426539, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:19.804118 27953 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:42:19.805111 27953 webserver.cc:460] Webserver started at http://127.10.224.131:35681/ using document root <none> and password file <none>
I20260926 12:42:19.805704 27953 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:19.805800 27953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:19.810505 27953 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 12:42:19.812879 27968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:19.814198 27953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:42:19.814364 27953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "17fe564b62434feead95d2ec809e166c"
format_stamp: "Formatted at 2026-09-26 12:41:37 on dist-test-slave-m3pj"
server_key: "af5d1e11825fe25c203b9be716fd91de"
server_key_iv: "e77ac0fe09392941d91755afaa1b6f40"
server_key_version: "encryptionkey@0"
I20260926 12:42:19.814965 27953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:19.832811 27953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:19.836025 27953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:19.836251 27953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:19.836867 27953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:19.837764 27953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:19.837836 27953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:19.837915 27953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:19.837970 27953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:19.847858 27953 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:38735
I20260926 12:42:19.847904 28081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:38735 every 8 connection(s)
I20260926 12:42:19.849013 27953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:42:19.853124 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:19.851278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:38517)
0926 12:42:19.851865 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.852367 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":147,"threads_started":1}
W20260926 12:42:19.853628 28082 heartbeater.cc:644] Failed to heartbeat to 127.10.224.190:34831 (0 consecutive failures): Network error: Failed to ping master at 127.10.224.190:34831: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
I20260926 12:42:19.854324 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:19.854045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:59449)
0926 12:42:19.854162 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.854213 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 12:42:19.854986 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:19.854746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:56477)
0926 12:42:19.854876 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:19.854924 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 12:42:19.855254 28082 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 12:42:19.858911 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 27953
I20260926 12:42:20.526207 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:20.525708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:50559)
0926 12:42:20.526037 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:20.526106 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
I20260926 12:42:20.664004 28087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:20.663598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:45671)
0926 12:42:20.663835 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:20.663898 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260926 12:42:20.856360 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:20.855913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:32815)
0926 12:42:20.856183 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:20.856255 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
I20260926 12:42:21.026815 28089 raft_consensus.cc:482] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:21.027017 28089 raft_consensus.cc:504] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:42:21.028328 28089 leader_election.cc:288] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
Sep 26 12:42:21 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426539, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.131@KRBTEST.COM
Sep 26 12:42:21 dist-test-slave-m3pj krb5kdc[26497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426539, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.130@KRBTEST.COM
I20260926 12:42:21.053223 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:21.029003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:41695 (local address 127.10.224.129:41421)
0926 12:42:21.029218 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:21.029231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:21.029325 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:21.031310 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:21.031325 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:21.032053 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:21.033038 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:21.033063 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:21.035356 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:21.035360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:21.037287 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:21.037303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:21.038531 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:21.038535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:21.038863 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:21.044376 (+ 5513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:21.044408 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:21.047202 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:21.050898 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:21.050900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:21.050901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:21.051101 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:21.051549 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:21.051552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:21.051555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:21.051641 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:21.052372 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:21.052375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:21.052486 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 12:42:21.052612 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 12:42:21.053007 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":505,"spinlock_wait_cycles":12160,"thread_start_us":64,"threads_started":1}
I20260926 12:42:21.053236 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:21.028690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:35903 (local address 127.10.224.131:38735)
0926 12:42:21.029154 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:21.029185 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:21.029239 (+ 54us) server_negotiation.cc:407] Connection header received
0926 12:42:21.030255 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:21.030265 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:21.030790 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:21.031168 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:21.033286 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:21.035104 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:21.037454 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:21.038304 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:21.038552 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:21.047164 (+ 8612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:21.047174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:21.047198 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:21.047249 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:21.050233 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:21.050932 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:21.050938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:21.050942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:21.051087 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:21.051490 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:21.051493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:21.051495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:21.052040 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:21.052192 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:21.052735 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 12:42:21.053042 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":61,"threads_started":1}
I20260926 12:42:21.053236 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:21.029189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:41421 (local address 127.10.224.130:41695)
0926 12:42:21.029540 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:21.029563 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:21.029599 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:42:21.030455 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:21.030463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:21.030792 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:21.031168 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:21.033286 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:21.035104 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:21.037454 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:21.038304 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:21.038526 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:21.047624 (+ 9098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:21.047631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:21.047656 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:21.047697 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:21.050232 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:21.051263 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:21.051269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:21.051272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:21.051406 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:21.051756 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:21.051759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:21.051761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:21.052177 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:21.052408 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:21.052776 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 12:42:21.053070 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":127,"threads_started":1}
I20260926 12:42:21.053268 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:21.028555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:38735 (local address 127.10.224.129:35903)
0926 12:42:21.028848 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:21.028861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:21.029001 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:21.031310 (+ 2309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:21.031318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:21.032053 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:21.033038 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:21.033063 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:21.035355 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:21.035360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:21.037287 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:21.037303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:21.038531 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:21.038535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:21.038838 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:21.044373 (+ 5535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:21.044425 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:21.046737 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:21.050421 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:21.050428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:21.050444 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:21.050794 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:21.051202 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:21.051206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:21.051208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:21.051369 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:21.052161 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:21.052168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:21.052394 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 12:42:21.052592 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 12:42:21.053100 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":542,"spinlock_wait_cycles":77184,"thread_start_us":109,"threads_started":1}
W20260926 12:42:21.056079 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
W20260926 12:42:21.056317 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
I20260926 12:42:21.056387 27703 leader_election.cc:302] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c7e1a2af6fa4922ae0586af5dd4ac2d; no voters: 17fe564b62434feead95d2ec809e166c, 2654cb4c5fe44695bec5770bd3585be5
I20260926 12:42:21.056674 28089 raft_consensus.cc:2726] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:42:21.527477 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:21.527147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:59089)
0926 12:42:21.527297 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:21.527357 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 12:42:21.665306 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:21.664849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:52077)
0926 12:42:21.665139 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:21.665205 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260926 12:42:21.857714 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:21.857238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:43053)
0926 12:42:21.857521 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:21.857574 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
I20260926 12:42:22.528961 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:22.528461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:55399)
0926 12:42:22.528795 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:22.528862 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260926 12:42:22.666812 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:22.666305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:44403)
0926 12:42:22.666608 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:22.666677 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":78,"threads_started":1}
I20260926 12:42:22.859125 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:22.858576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:40481)
0926 12:42:22.858941 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:22.859007 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":109,"threads_started":1}
I20260926 12:42:23.255568 28099 raft_consensus.cc:482] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:23.255728 28099 raft_consensus.cc:504] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:42:23.256669 28099 leader_election.cc:288] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
W20260926 12:42:23.257273 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
W20260926 12:42:23.257561 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
I20260926 12:42:23.257700 27703 leader_election.cc:302] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c7e1a2af6fa4922ae0586af5dd4ac2d; no voters: 17fe564b62434feead95d2ec809e166c, 2654cb4c5fe44695bec5770bd3585be5
I20260926 12:42:23.257903 28099 raft_consensus.cc:2726] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:42:23.530221 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:23.529790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:54931)
0926 12:42:23.530057 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:23.530122 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260926 12:42:23.668220 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:23.667768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:49291)
0926 12:42:23.668049 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:23.668119 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260926 12:42:23.860538 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:23.859997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:33181)
0926 12:42:23.860343 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:23.860436 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":88,"threads_started":1}
I20260926 12:42:24.531731 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:24.531181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:46965)
0926 12:42:24.531502 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:24.531630 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260926 12:42:24.669556 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:24.669052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:45129)
0926 12:42:24.669368 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:24.669435 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20260926 12:42:24.861874 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:24.861418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:54919)
0926 12:42:24.861702 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:24.861770 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260926 12:42:25.533027 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:25.532549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:46313)
0926 12:42:25.532844 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:25.532898 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":130,"threads_started":1}
I20260926 12:42:25.670855 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:25.670419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:46905)
0926 12:42:25.670691 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:25.670756 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260926 12:42:25.863358 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:25.862840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:56595)
0926 12:42:25.863150 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:25.863219 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":108,"threads_started":1}
I20260926 12:42:26.157411 28109 raft_consensus.cc:482] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:26.157564 28109 raft_consensus.cc:504] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:42:26.158085 28109 leader_election.cc:288] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
W20260926 12:42:26.159047 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
W20260926 12:42:26.159271 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
I20260926 12:42:26.159343 27703 leader_election.cc:302] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c7e1a2af6fa4922ae0586af5dd4ac2d; no voters: 17fe564b62434feead95d2ec809e166c, 2654cb4c5fe44695bec5770bd3585be5
I20260926 12:42:26.159571 28109 raft_consensus.cc:2726] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:42:26.534319 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:26.533864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:54359)
0926 12:42:26.534151 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:26.534215 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260926 12:42:26.672439 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:26.671914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:40251)
0926 12:42:26.672233 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:26.672300 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":72,"threads_started":1}
I20260926 12:42:26.864749 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:26.864278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:59045)
0926 12:42:26.864593 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:26.864654 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":73,"threads_started":1}
I20260926 12:42:27.535708 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:27.535163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:58991)
0926 12:42:27.535477 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:27.535603 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":78,"threads_started":1}
I20260926 12:42:27.673807 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:27.673339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:33123)
0926 12:42:27.673639 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:27.673705 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
I20260926 12:42:27.866101 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:27.865573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:57983)
0926 12:42:27.865917 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:27.865988 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260926 12:42:28.536966 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:28.536514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:42597)
0926 12:42:28.536783 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:28.536845 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260926 12:42:28.675202 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:28.674707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:55467)
0926 12:42:28.675027 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:28.675097 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
I20260926 12:42:28.867363 28118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:28.866936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:47305)
0926 12:42:28.867196 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:28.867255 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260926 12:42:29.538372 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:29.537939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:44365)
0926 12:42:29.538203 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:29.538266 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260926 12:42:29.676550 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:29.676098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:46307)
0926 12:42:29.676359 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:29.676453 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260926 12:42:29.699676 28121 raft_consensus.cc:482] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:29.699858 28121 raft_consensus.cc:504] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:42:29.700295 28121 leader_election.cc:288] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
W20260926 12:42:29.701159 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
W20260926 12:42:29.701318 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
I20260926 12:42:29.701378 27703 leader_election.cc:302] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c7e1a2af6fa4922ae0586af5dd4ac2d; no voters: 17fe564b62434feead95d2ec809e166c, 2654cb4c5fe44695bec5770bd3585be5
I20260926 12:42:29.701582 28121 raft_consensus.cc:2726] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:42:29.868795 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:29.868354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:48039)
0926 12:42:29.868637 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:29.868699 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260926 12:42:30.539856 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:30.539350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:59339)
0926 12:42:30.539689 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:30.539756 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
I20260926 12:42:30.677996 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:30.677526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:39369)
0926 12:42:30.677800 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:30.677868 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260926 12:42:30.870208 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:30.869736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:46699)
0926 12:42:30.870050 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:30.870113 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20260926 12:42:31.541296 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:31.540855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:43743)
0926 12:42:31.541133 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:31.541197 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260926 12:42:31.679294 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:31.678814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:35759)
0926 12:42:31.679134 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:31.679196 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260926 12:42:31.871825 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:31.871273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:46729)
0926 12:42:31.871640 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:31.871709 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
I20260926 12:42:32.412168 28129 raft_consensus.cc:482] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:32.412305 28129 raft_consensus.cc:504] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17fe564b62434feead95d2ec809e166c" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 38735 } } peers { permanent_uuid: "2654cb4c5fe44695bec5770bd3585be5" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 41695 } } peers { permanent_uuid: "9c7e1a2af6fa4922ae0586af5dd4ac2d" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 39179 } }
I20260926 12:42:32.412724 28129 leader_election.cc:288] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735), 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695)
W20260926 12:42:32.413745 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2654cb4c5fe44695bec5770bd3585be5 (127.10.224.130:41695): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
W20260926 12:42:32.413944 27703 leader_election.cc:341] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 17fe564b62434feead95d2ec809e166c (127.10.224.131:38735): Not found: Tablet not found: 7165f274c2f64134bb3f72920a1b7b7d
I20260926 12:42:32.414021 27703 leader_election.cc:302] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c7e1a2af6fa4922ae0586af5dd4ac2d; no voters: 17fe564b62434feead95d2ec809e166c, 2654cb4c5fe44695bec5770bd3585be5
I20260926 12:42:32.414212 28129 raft_consensus.cc:2726] T 7165f274c2f64134bb3f72920a1b7b7d P 9c7e1a2af6fa4922ae0586af5dd4ac2d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:42:32.542623 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:32.542144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:38929)
0926 12:42:32.542466 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:32.542525 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260926 12:42:32.680720 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:32.680226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:51607)
0926 12:42:32.680548 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:32.680619 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260926 12:42:32.873188 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:32.872692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:38017)
0926 12:42:32.873004 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:32.873086 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":79,"threads_started":1}
I20260926 12:42:33.543998 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:33.543515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:39207)
0926 12:42:33.543831 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:33.543888 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260926 12:42:33.682116 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:33.681663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:56949)
0926 12:42:33.681951 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:33.682017 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260926 12:42:33.874754 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:33.874220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:47505)
0926 12:42:33.874570 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:33.874642 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":127,"threads_started":1}
I20260926 12:42:34.545428 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:34.545017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.129:45517)
0926 12:42:34.545269 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:34.545329 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260926 12:42:34.683373 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:34.682954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.130:51527)
0926 12:42:34.683217 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:34.683270 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
I20260926 12:42:34.869407 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 27646
I20260926 12:42:34.875416 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 27819
I20260926 12:42:34.876803 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:34.876246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34831 (local address 127.10.224.131:50253)
0926 12:42:34.876606 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:34.876672 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.190:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":67,"threads_started":1}
I20260926 12:42:34.881134 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 27953
2026-09-26T12:42:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59375 ms)
[----------] 30 tests from SecurityITest (256794 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:34 dist-test-slave-m3pj krb5kdc[28141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:34 dist-test-slave-m3pj krb5kdc[28141](info): set up 2 sockets
Sep 26 12:42:34 dist-test-slave-m3pj krb5kdc[28141](info): commencing operation
krb5kdc: starting...
W20260926 12:42:35.951887 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:36 dist-test-slave-m3pj krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:42:36.088471 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:42463
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:42463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:36.205585 28154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:36.205948 28154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:36.206072 28154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:36.210425 28154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:36.210551 28154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:36.210592 28154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:36.210633 28154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:36.210670 28154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:36.216512 28154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:42463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:36.217774 28154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:36.218979 28154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:36.228392 28160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:36.228379 28162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:36.228587 28154 server_base.cc:1034] running on GCE node
W20260926 12:42:36.228399 28159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:36.229413 28154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:36.230739 28154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:36.231976 28154 hybrid_clock.cc:648] HybridClock initialized: now 1790426556231954 us; error 38 us; skew 500 ppm
Sep 26 12:42:36 dist-test-slave-m3pj krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:36.235085 28154 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:42:36.236544 28154 webserver.cc:460] Webserver started at http://127.10.224.190:35579/ using document root <none> and password file <none>
I20260926 12:42:36.237267 28154 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:36.237387 28154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:36.237650 28154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:36.240203 28154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "fc286e23307940a590c711643c4ab24c"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "63bd0af3266da716757cd68647665678"
server_key_iv: "d91213a17345080ad5b9f31350cce28c"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.240965 28154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "fc286e23307940a590c711643c4ab24c"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "63bd0af3266da716757cd68647665678"
server_key_iv: "d91213a17345080ad5b9f31350cce28c"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.245844 28154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 12:42:36.248549 28169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:36.249910 28154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:42:36.250063 28154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "fc286e23307940a590c711643c4ab24c"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "63bd0af3266da716757cd68647665678"
server_key_iv: "d91213a17345080ad5b9f31350cce28c"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.250190 28154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:36.271461 28154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:36.275014 28154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:36.275194 28154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:36.282828 28154 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:42463
I20260926 12:42:36.282850 28221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:42463 every 8 connection(s)
I20260926 12:42:36.283910 28154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:36.285969 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 28154
I20260926 12:42:36.286096 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:36.286441 11138 external_mini_cluster.cc:1428] Setting key 499720d90c478d3c5f56fcac6d4c7c52
I20260926 12:42:36.287221 28222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:36.293362 28222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c: Bootstrap starting.
Sep 26 12:42:36 dist-test-slave-m3pj krb5kdc[28141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:36.296044 28222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:36.297060 28222 log.cc:824] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:36.299113 28222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c: No bootstrap required, opened a new log
I20260926 12:42:36.301093 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:36.288013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.10.224.190:42463)
0926 12:42:36.288484 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:36.288499 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:36.288536 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:42:36.289196 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:36.289213 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:36.289563 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:36.289916 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:36.290768 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.291833 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.292690 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.293177 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.293351 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:36.295981 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:36.295991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:36.296011 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:36.296050 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:36.297850 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:36.298327 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:36.298332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:36.298337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:36.298454 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:36.298745 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:36.298749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:36.298751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:36.299193 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:36.299400 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:36.299754 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 12:42:36.300111 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":156,"threads_started":1}
I20260926 12:42:36.302585 28222 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc286e23307940a590c711643c4ab24c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42463 } }
I20260926 12:42:36.302878 28222 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:36.302953 28222 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc286e23307940a590c711643c4ab24c, State: Initialized, Role: FOLLOWER
I20260926 12:42:36.303479 28222 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc286e23307940a590c711643c4ab24c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42463 } }
I20260926 12:42:36.303683 28222 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:36.303772 28222 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:36.303879 28222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:36.304867 28222 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc286e23307940a590c711643c4ab24c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42463 } }
I20260926 12:42:36.305218 28222 leader_election.cc:302] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc286e23307940a590c711643c4ab24c; no voters:
I20260926 12:42:36.305536 28222 leader_election.cc:288] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:36.305687 28227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:36.305959 28227 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [term 1 LEADER]: Becoming Leader. State: Replica: fc286e23307940a590c711643c4ab24c, State: Running, Role: LEADER
I20260926 12:42:36.306262 28227 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc286e23307940a590c711643c4ab24c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42463 } }
I20260926 12:42:36.306872 28222 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:36.308219 28228 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc286e23307940a590c711643c4ab24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc286e23307940a590c711643c4ab24c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42463 } } }
I20260926 12:42:36.308350 28228 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:36.308665 28229 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc286e23307940a590c711643c4ab24c. Latest consensus state: current_term: 1 leader_uuid: "fc286e23307940a590c711643c4ab24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc286e23307940a590c711643c4ab24c" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42463 } } }
I20260926 12:42:36.308750 28229 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:36.308869 28236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:36.311720 28236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:36.318550 28236 catalog_manager.cc:1347] Generated new cluster ID: 7f857b094cb84337839a2f35c6e2f381
I20260926 12:42:36.318661 28236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:36.336655 28236 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:36.337729 28236 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:36.347306 28236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c: Generated new TSK 0
I20260926 12:42:36.348212 28236 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:42:36.384698 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42463
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:36.503545 28248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:36.503890 28248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:36.503942 28248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:36.508374 28248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:36.508446 28248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:36.508554 28248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:42:36.514272 28248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:36.515486 28248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:36.516690 28248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:36.527354 28256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:36.527386 28253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:36.527423 28248 server_base.cc:1034] running on GCE node
W20260926 12:42:36.527658 28254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:36.528144 28248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:36.528869 28248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:36.530061 28248 hybrid_clock.cc:648] HybridClock initialized: now 1790426556530027 us; error 48 us; skew 500 ppm
Sep 26 12:42:36 dist-test-slave-m3pj krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:36.533010 28248 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:42:36.533918 28248 webserver.cc:460] Webserver started at http://127.10.224.129:33519/ using document root <none> and password file <none>
I20260926 12:42:36.534493 28248 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:36.534585 28248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:36.534825 28248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:36.537012 28248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "ca885675244c4f6a9030080d1751f293"
server_key_iv: "4bcfc55d84432d304382f1c6830b9103"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.537614 28248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "ca885675244c4f6a9030080d1751f293"
server_key_iv: "4bcfc55d84432d304382f1c6830b9103"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.541462 28248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:42:36.543988 28263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:36.545276 28248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:42:36.545426 28248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "ca885675244c4f6a9030080d1751f293"
server_key_iv: "4bcfc55d84432d304382f1c6830b9103"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.545553 28248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:36.569850 28248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:36.573355 28248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:36.573633 28248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:36.574285 28248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:36.575191 28248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:36.575264 28248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:36.575336 28248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:36.575380 28248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:36.585029 28248 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:38117
I20260926 12:42:36.585048 28376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:38117 every 8 connection(s)
I20260926 12:42:36.586030 28248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:42:36.592028 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 28248
I20260926 12:42:36.592185 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:42:36.592813 11138 external_mini_cluster.cc:1428] Setting key e0a27c5f0e666540ba1a22273d7bd8b9
Sep 26 12:42:36 dist-test-slave-m3pj krb5kdc[28141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:36.602945 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:36.588102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:46227 (local address 127.10.224.190:42463)
0926 12:42:36.588263 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:36.588267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:36.589303 (+ 1036us) server_negotiation.cc:407] Connection header received
0926 12:42:36.590924 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:36.590927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:36.590983 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:36.591084 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:36.592767 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.593534 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.594864 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.595123 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.595243 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:36.598385 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:36.598388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:36.598391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:36.598429 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:36.600057 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:36.600932 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:36.600935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:36.600936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:36.601032 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:36.601512 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:36.601515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:36.601516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:36.601729 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:36.601827 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:36.602662 (+ 835us) server_negotiation.cc:299] Negotiation successful
0926 12:42:36.602796 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:42:36.604225 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:36.588521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42463 (local address 127.10.224.129:46227)
0926 12:42:36.589081 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:36.589150 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:36.590703 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:36.591295 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:36.591303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:36.591923 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:36.592568 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:36.592584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.593685 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.593687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:36.594731 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:36.594742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.595292 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.595296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:36.595488 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:36.596204 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:36.596222 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:36.598149 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:36.600231 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:36.600242 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:36.600270 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:36.600751 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:36.601153 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:36.601158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:36.601163 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:36.601401 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:36.601862 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:36.601872 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:36.602314 (+ 442us) client_negotiation.cc:769] Sending connection context
0926 12:42:36.602572 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 12:42:36.603035 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260926 12:42:36.605636 28377 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42463
I20260926 12:42:36.605937 28377 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:42:36.606662 28377 heartbeater.cc:511] Master 127.10.224.190:42463 requested a full tablet report, sending...
I20260926 12:42:36.608824 28186 ts_manager.cc:194] Registered new tserver with Master: 9df54440aa2d4c0fa6ffe9f751d37fc4 (127.10.224.129:38117)
I20260926 12:42:36.610101 28186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:46227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:42:36.627386 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42463
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:36.743839 28382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:36.744171 28382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:36.744256 28382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:36.748574 28382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:36.748665 28382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:36.748772 28382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:42:36.754555 28382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:36.755807 28382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:36.756868 28382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:36.767017 28390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:36.767017 28387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:36.767074 28388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:36.767417 28382 server_base.cc:1034] running on GCE node
I20260926 12:42:36.767891 28382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:36.768654 28382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:36.769860 28382 hybrid_clock.cc:648] HybridClock initialized: now 1790426556769841 us; error 38 us; skew 500 ppm
Sep 26 12:42:36 dist-test-slave-m3pj krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:36.772843 28382 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:42:36.773810 28382 webserver.cc:460] Webserver started at http://127.10.224.130:41541/ using document root <none> and password file <none>
I20260926 12:42:36.774355 28382 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:36.774415 28382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:36.774581 28382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:36.776736 28382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "cd106e7c31a84d42abdf3aadfce5b08e"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "ae6a43835b080c39eb937ec709a96c0d"
server_key_iv: "9be223de1bbfa9ef3141bca555c72a10"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.777264 28382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "cd106e7c31a84d42abdf3aadfce5b08e"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "ae6a43835b080c39eb937ec709a96c0d"
server_key_iv: "9be223de1bbfa9ef3141bca555c72a10"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.781232 28382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 12:42:36.783869 28397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:36.785156 28382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:42:36.785300 28382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "cd106e7c31a84d42abdf3aadfce5b08e"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "ae6a43835b080c39eb937ec709a96c0d"
server_key_iv: "9be223de1bbfa9ef3141bca555c72a10"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.785454 28382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:36.797369 28382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:36.800876 28382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:36.801095 28382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:36.801739 28382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:36.802662 28382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:36.802714 28382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:36.802784 28382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:36.802826 28382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:36.812732 28382 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:32873
I20260926 12:42:36.812776 28510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:32873 every 8 connection(s)
I20260926 12:42:36.813756 28382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:42:36.814270 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 28382
I20260926 12:42:36.814420 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:42:36.814750 11138 external_mini_cluster.cc:1428] Setting key 844069a971222613c1b954ed23834627
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
Sep 26 12:42:36 dist-test-slave-m3pj krb5kdc[28141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:36.834842 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:36.815888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:46719 (local address 127.10.224.190:42463)
0926 12:42:36.816026 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:36.816030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:36.817117 (+ 1087us) server_negotiation.cc:407] Connection header received
0926 12:42:36.818774 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:36.818777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:36.818824 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:36.818914 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:36.820682 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.821297 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.823295 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.823648 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.823761 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:36.830125 (+ 6364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:36.830129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:36.830131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:36.830162 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:36.832135 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:36.832783 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:36.832790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:36.832795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:36.832899 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:36.833349 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:36.833356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:36.833361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:36.833670 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:36.833786 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:36.834405 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 12:42:36.834589 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:42:36.835832 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:36.816435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42463 (local address 127.10.224.130:46719)
0926 12:42:36.816975 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:36.817019 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:36.818507 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:36.819165 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:36.819176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:36.819750 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:36.820505 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:36.820531 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.821440 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.821446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:36.823134 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:36.823150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:36.823831 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:36.823838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:36.824082 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:36.824807 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:36.824835 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:36.829875 (+ 5040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:36.832311 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:36.832317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:36.832334 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:36.832653 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:36.833032 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:36.833035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:36.833036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:36.833217 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:36.833801 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:36.833808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:36.834114 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 12:42:36.834297 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 12:42:36.834625 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":155,"threads_started":1}
I20260926 12:42:36.837239 28511 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42463
I20260926 12:42:36.837565 28511 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:36.838106 28511 heartbeater.cc:511] Master 127.10.224.190:42463 requested a full tablet report, sending...
I20260926 12:42:36.839365 28186 ts_manager.cc:194] Registered new tserver with Master: cd106e7c31a84d42abdf3aadfce5b08e (127.10.224.130:32873)
I20260926 12:42:36.840152 28186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:46719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:42:36.847733 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42463
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:36.963027 28516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:36.963397 28516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:36.963514 28516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:36.967778 28516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:36.967885 28516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:36.967983 28516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:42:36.973718 28516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:36.975296 28516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:36.976501 28516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:36.987033 28521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:36.987054 28524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:36.987074 28522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:36.987334 28516 server_base.cc:1034] running on GCE node
I20260926 12:42:36.987874 28516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:36.988654 28516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:36.989845 28516 hybrid_clock.cc:648] HybridClock initialized: now 1790426556989826 us; error 37 us; skew 500 ppm
Sep 26 12:42:36 dist-test-slave-m3pj krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:36.992841 28516 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:42:36.993806 28516 webserver.cc:460] Webserver started at http://127.10.224.131:46509/ using document root <none> and password file <none>
I20260926 12:42:36.994478 28516 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:36.994549 28516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:36.994722 28516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:36.996982 28516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "0b3cc6be6dd74fbfba6d309131fbaebe"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "1ae442ebecef82358ef87644c08c3a5c"
server_key_iv: "06ea53077dc823f28521984799631f54"
server_key_version: "encryptionkey@0"
I20260926 12:42:36.997547 28516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "0b3cc6be6dd74fbfba6d309131fbaebe"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "1ae442ebecef82358ef87644c08c3a5c"
server_key_iv: "06ea53077dc823f28521984799631f54"
server_key_version: "encryptionkey@0"
I20260926 12:42:37.001863 28516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:42:37.004871 28531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:37.006230 28516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:42:37.006337 28516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "0b3cc6be6dd74fbfba6d309131fbaebe"
format_stamp: "Formatted at 2026-09-26 12:42:36 on dist-test-slave-m3pj"
server_key: "1ae442ebecef82358ef87644c08c3a5c"
server_key_iv: "06ea53077dc823f28521984799631f54"
server_key_version: "encryptionkey@0"
I20260926 12:42:37.006497 28516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:37.022552 28516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:37.026528 28516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:37.026747 28516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:37.027400 28516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:37.028419 28516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:37.028469 28516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:37.028532 28516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:37.028566 28516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:37.038060 28516 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:33739
I20260926 12:42:37.038077 28644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:33739 every 8 connection(s)
I20260926 12:42:37.039078 28516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:42:37.044993 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 28516
I20260926 12:42:37.045156 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:42:37.045511 11138 external_mini_cluster.cc:1428] Setting key 30ce68c1c6c5a81fa4d25c6eeaa61076
Sep 26 12:42:37 dist-test-slave-m3pj krb5kdc[28141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:37.055711 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.041158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:44839 (local address 127.10.224.190:42463)
0926 12:42:37.041296 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.041300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.042322 (+ 1022us) server_negotiation.cc:407] Connection header received
0926 12:42:37.043716 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.043720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.043786 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.043878 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:37.045412 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.046177 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.047872 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.048194 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.048291 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.051330 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:37.051334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:37.051336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:37.051370 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:37.053156 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:37.053740 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:37.053742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:37.053744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:37.053830 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:37.054246 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:37.054248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:37.054249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:37.054566 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:37.054654 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.055366 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.055556 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:42:37.056687 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.041591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42463 (local address 127.10.224.131:44839)
0926 12:42:37.042116 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.042163 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:37.043420 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:37.044074 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:37.044081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:37.044478 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:37.045188 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.045213 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.046344 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.046347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.047718 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.047746 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.048369 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.048373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.048575 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.049346 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:37.049365 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:37.051111 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:37.053295 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:37.053302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:37.053319 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:37.053633 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:37.053919 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:37.053922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:37.053923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:37.054153 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:37.054700 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:37.054707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:37.055043 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 12:42:37.055278 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 12:42:37.055726 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260926 12:42:37.058048 28645 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42463
I20260926 12:42:37.058420 28645 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:37.059125 28645 heartbeater.cc:511] Master 127.10.224.190:42463 requested a full tablet report, sending...
I20260926 12:42:37.060391 28186 ts_manager.cc:194] Registered new tserver with Master: 0b3cc6be6dd74fbfba6d309131fbaebe (127.10.224.131:33739)
I20260926 12:42:37.061057 28186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:44839
I20260926 12:42:37.072749 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:42:37.082612 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.074902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.10.224.190:42463)
0926 12:42:37.075112 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.075115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.075163 (+ 48us) server_negotiation.cc:407] Connection header received
0926 12:42:37.075296 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.075299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.075343 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.075456 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:37.076222 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.076786 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.077597 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.077813 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.077899 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.079424 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:37.079427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:37.079429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:37.079456 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:37.080986 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:37.081397 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:37.081399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:37.081401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:37.081482 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:37.081707 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:37.081709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:37.081710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:37.081930 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:37.082036 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.082247 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.082362 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 12:42:37.086103 28186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:42:37.089532 28186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:42:37.106352 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.099567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:32873 (local address 127.0.0.1:47892)
0926 12:42:37.100263 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.100347 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:37.100509 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:37.101422 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:37.101428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:37.101455 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:37.101602 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.101609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.103305 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.103308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.104834 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.104845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.105801 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.105805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.105912 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.105949 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 12:42:37.106069 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 12:42:37.106202 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":81,"thread_start_us":97,"threads_started":1}
I20260926 12:42:37.106359 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.099664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.10.224.130:32873)
0926 12:42:37.100028 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.100043 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.100679 (+ 636us) server_negotiation.cc:407] Connection header received
0926 12:42:37.100776 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.100781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.101025 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.101196 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:37.101776 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.103159 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.104977 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.105665 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.105822 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.105980 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.106096 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.106195 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":157,"threads_started":1}
I20260926 12:42:37.106608 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.100525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:33739 (local address 127.0.0.1:38916)
0926 12:42:37.101002 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.101014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:37.101101 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:37.101728 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:37.101733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:37.101749 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:37.101887 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.101894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.103441 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.103445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.104750 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.104768 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.106123 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.106130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.106254 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.106280 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:42:37.106367 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:42:37.106445 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":18,"thread_start_us":95,"threads_started":1}
I20260926 12:42:37.107270 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.100818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.10.224.131:33739)
0926 12:42:37.101275 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.101286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.101310 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:42:37.101390 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.101396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.101580 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.101746 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:37.102012 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.103241 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.104892 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.105773 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.105911 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.106044 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.106723 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.106842 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":146,"threads_started":1}
I20260926 12:42:37.107010 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.099634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:38117 (local address 127.0.0.1:44466)
0926 12:42:37.101234 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.101245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:37.101390 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:37.102015 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:37.102019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:37.102033 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:37.102110 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.102114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.103167 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.103172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.105096 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.105104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.106502 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.106507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.106609 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.106645 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 12:42:37.106792 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 12:42:37.106874 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":493,"thread_start_us":89,"threads_started":1}
I20260926 12:42:37.108289 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.100453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.10.224.129:38117)
0926 12:42:37.100853 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.100869 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.101321 (+ 452us) server_negotiation.cc:407] Connection header received
0926 12:42:37.101540 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.101544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.101719 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.101864 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:37.102219 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.103041 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.105224 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.106167 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.106297 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.106437 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.107971 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.108150 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":74,"threads_started":1}
I20260926 12:42:37.110246 28579 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed9fabdf21e461f97ab9ade3060754b (DEFAULT_TABLE table=test-table [id=e2bbf4befc3948c4b62714566e13b15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:37.110251 28445 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed9fabdf21e461f97ab9ade3060754b (DEFAULT_TABLE table=test-table [id=e2bbf4befc3948c4b62714566e13b15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:37.110869 28311 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed9fabdf21e461f97ab9ade3060754b (DEFAULT_TABLE table=test-table [id=e2bbf4befc3948c4b62714566e13b15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:37.111359 28579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed9fabdf21e461f97ab9ade3060754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:37.111356 28445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed9fabdf21e461f97ab9ade3060754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:37.111837 28311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed9fabdf21e461f97ab9ade3060754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:37.118594 28662 tablet_bootstrap.cc:492] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Bootstrap starting.
I20260926 12:42:37.119905 28664 tablet_bootstrap.cc:492] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Bootstrap starting.
I20260926 12:42:37.121968 28664 tablet_bootstrap.cc:654] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:37.122179 28662 tablet_bootstrap.cc:654] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:37.123198 28664 log.cc:824] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:37.123350 28662 log.cc:824] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:37.123905 28663 tablet_bootstrap.cc:492] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4: Bootstrap starting.
I20260926 12:42:37.125530 28664 tablet_bootstrap.cc:492] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: No bootstrap required, opened a new log
I20260926 12:42:37.125602 28662 tablet_bootstrap.cc:492] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: No bootstrap required, opened a new log
I20260926 12:42:37.125782 28664 ts_tablet_manager.cc:1397] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 12:42:37.125788 28662 ts_tablet_manager.cc:1397] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:42:37.126281 28663 tablet_bootstrap.cc:654] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:37.127157 28663 log.cc:824] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:37.129494 28663 tablet_bootstrap.cc:492] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4: No bootstrap required, opened a new log
I20260926 12:42:37.129422 28662 raft_consensus.cc:348] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
I20260926 12:42:37.129735 28663 ts_tablet_manager.cc:1397] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 12:42:37.129737 28662 raft_consensus.cc:374] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:37.129828 28662 raft_consensus.cc:729] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3cc6be6dd74fbfba6d309131fbaebe, State: Initialized, Role: FOLLOWER
I20260926 12:42:37.130308 28662 consensus_queue.cc:260] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
I20260926 12:42:37.130364 28664 raft_consensus.cc:348] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
I20260926 12:42:37.130648 28664 raft_consensus.cc:374] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:37.130723 28664 raft_consensus.cc:729] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd106e7c31a84d42abdf3aadfce5b08e, State: Initialized, Role: FOLLOWER
I20260926 12:42:37.131201 28664 consensus_queue.cc:260] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
I20260926 12:42:37.131376 28662 ts_tablet_manager.cc:1428] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:42:37.131857 28645 heartbeater.cc:503] Master 127.10.224.190:42463 was elected leader, sending a full tablet report...
I20260926 12:42:37.131912 28511 heartbeater.cc:503] Master 127.10.224.190:42463 was elected leader, sending a full tablet report...
I20260926 12:42:37.134843 28663 raft_consensus.cc:348] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
I20260926 12:42:37.135193 28663 raft_consensus.cc:374] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:37.135254 28663 raft_consensus.cc:729] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df54440aa2d4c0fa6ffe9f751d37fc4, State: Initialized, Role: FOLLOWER
I20260926 12:42:37.135812 28663 consensus_queue.cc:260] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
I20260926 12:42:37.136965 28663 ts_tablet_manager.cc:1428] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260926 12:42:37.136991 28377 heartbeater.cc:503] Master 127.10.224.190:42463 was elected leader, sending a full tablet report...
I20260926 12:42:37.138067 28664 ts_tablet_manager.cc:1428] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Time spent starting tablet: real 0.012s user 0.000s sys 0.008s
I20260926 12:42:37.274530 28670 raft_consensus.cc:482] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:37.274752 28670 raft_consensus.cc:504] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
I20260926 12:42:37.276103 28670 leader_election.cc:288] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd106e7c31a84d42abdf3aadfce5b08e (127.10.224.130:32873), 0b3cc6be6dd74fbfba6d309131fbaebe (127.10.224.131:33739)
I20260926 12:42:37.282171 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.276464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:33441 (local address 127.10.224.130:32873)
0926 12:42:37.276616 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.276623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.276920 (+ 297us) server_negotiation.cc:407] Connection header received
0926 12:42:37.277068 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.277072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.277130 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.277265 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:37.277635 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.278465 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.280546 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.281263 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.281426 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.281489 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.281928 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.282027 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 12:42:37.282171 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.276508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:39697 (local address 127.10.224.131:33739)
0926 12:42:37.276642 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.276646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.276977 (+ 331us) server_negotiation.cc:407] Connection header received
0926 12:42:37.277068 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.277071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.277128 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.277236 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:37.277635 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.278500 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.280546 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.281263 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.281426 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.281489 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.281928 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.282027 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:42:37.282261 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.276348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:32873 (local address 127.10.224.129:33441)
0926 12:42:37.276791 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.276809 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:37.276954 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:37.277283 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:37.277287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:37.277338 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:37.277494 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.277501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.278639 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.278643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.280359 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.280374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.281457 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.281461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.281705 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.281770 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 12:42:37.281864 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:42:37.282013 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":283,"spinlock_wait_cycles":5632,"thread_start_us":144,"threads_started":1}
I20260926 12:42:37.282261 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.276424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:33739 (local address 127.10.224.129:39697)
0926 12:42:37.276867 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.276879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:37.276968 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:37.277283 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:37.277287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:37.277336 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:37.277494 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.277501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.278639 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.278643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.280359 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.280374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.281457 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.281461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.281702 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.281766 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 12:42:37.281861 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 12:42:37.282013 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":196,"spinlock_wait_cycles":39040,"thread_start_us":116,"threads_started":1}
I20260926 12:42:37.283124 28465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed9fabdf21e461f97ab9ade3060754b" candidate_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" is_pre_election: true
I20260926 12:42:37.283489 28465 raft_consensus.cc:2454] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df54440aa2d4c0fa6ffe9f751d37fc4 in term 0.
I20260926 12:42:37.283905 28599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed9fabdf21e461f97ab9ade3060754b" candidate_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" is_pre_election: true
I20260926 12:42:37.284161 28267 leader_election.cc:302] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9df54440aa2d4c0fa6ffe9f751d37fc4, cd106e7c31a84d42abdf3aadfce5b08e; no voters:
I20260926 12:42:37.284235 28599 raft_consensus.cc:2454] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9df54440aa2d4c0fa6ffe9f751d37fc4 in term 0.
I20260926 12:42:37.284509 28670 raft_consensus.cc:2781] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:42:37.284628 28670 raft_consensus.cc:482] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:42:37.284670 28670 raft_consensus.cc:3037] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:37.286059 28670 raft_consensus.cc:504] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
I20260926 12:42:37.286491 28670 leader_election.cc:288] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [CANDIDATE]: Term 1 election: Requested vote from peers cd106e7c31a84d42abdf3aadfce5b08e (127.10.224.130:32873), 0b3cc6be6dd74fbfba6d309131fbaebe (127.10.224.131:33739)
I20260926 12:42:37.286923 28465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed9fabdf21e461f97ab9ade3060754b" candidate_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd106e7c31a84d42abdf3aadfce5b08e"
I20260926 12:42:37.286945 28599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed9fabdf21e461f97ab9ade3060754b" candidate_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe"
I20260926 12:42:37.287077 28599 raft_consensus.cc:3037] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:37.287114 28465 raft_consensus.cc:3037] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:37.288358 28465 raft_consensus.cc:2454] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df54440aa2d4c0fa6ffe9f751d37fc4 in term 1.
I20260926 12:42:37.288463 28599 raft_consensus.cc:2454] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9df54440aa2d4c0fa6ffe9f751d37fc4 in term 1.
I20260926 12:42:37.288796 28267 leader_election.cc:302] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9df54440aa2d4c0fa6ffe9f751d37fc4, cd106e7c31a84d42abdf3aadfce5b08e; no voters:
I20260926 12:42:37.289062 28670 raft_consensus.cc:2781] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:37.289337 28670 raft_consensus.cc:686] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [term 1 LEADER]: Becoming Leader. State: Replica: 9df54440aa2d4c0fa6ffe9f751d37fc4, State: Running, Role: LEADER
I20260926 12:42:37.289764 28670 consensus_queue.cc:237] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } }
W20260926 12:42:37.290117 28646 tablet.cc:2369] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:37.293777 28186 catalog_manager.cc:5495] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9df54440aa2d4c0fa6ffe9f751d37fc4 (127.10.224.129). New cstate: current_term: 1 leader_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9df54440aa2d4c0fa6ffe9f751d37fc4" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 38117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 } health_report { overall_health: UNKNOWN } } }
W20260926 12:42:37.315119 28512 tablet.cc:2369] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:42:37.337499 28378 tablet.cc:2369] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:37.349448 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.345610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.10.224.129:38117)
0926 12:42:37.345742 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.345746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.345759 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:42:37.345857 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.345859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.345915 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.346010 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:42:37.346328 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.346847 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.347795 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.348009 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.348089 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.348419 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:42:37.349019 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:42:37.349116 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.349194 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.349263 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:42:37.356323 28465 raft_consensus.cc:1261] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 1 FOLLOWER]: Refusing update from remote peer 9df54440aa2d4c0fa6ffe9f751d37fc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:37.356323 28599 raft_consensus.cc:1261] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 1 FOLLOWER]: Refusing update from remote peer 9df54440aa2d4c0fa6ffe9f751d37fc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:37.357154 28670 consensus_queue.cc:1035] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd106e7c31a84d42abdf3aadfce5b08e" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:37.357389 28673 consensus_queue.cc:1035] T 2ed9fabdf21e461f97ab9ade3060754b P 9df54440aa2d4c0fa6ffe9f751d37fc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b3cc6be6dd74fbfba6d309131fbaebe" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 33739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:37.368845 28677 mvcc.cc:205] Tried to move back new op lower bound from 7333587178923003904 to 7333587178661982208. Current Snapshot: MvccSnapshot[applied={T|T < 7333587178923003904 or (T in {7333587178923003904})}]
I20260926 12:42:37.375962 28185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:37.376170 28185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:37.378829 28185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc286e23307940a590c711643c4ab24c: Sending DeleteTablet for 3 replicas of tablet 2ed9fabdf21e461f97ab9ade3060754b
I20260926 12:42:37.379839 28311 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed9fabdf21e461f97ab9ade3060754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:37 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:44466
I20260926 12:42:37.379871 28579 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed9fabdf21e461f97ab9ade3060754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:37 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:38916
I20260926 12:42:37.380652 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 28248
I20260926 12:42:37.380635 28445 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed9fabdf21e461f97ab9ade3060754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:37 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:47892
I20260926 12:42:37.382743 28688 tablet_replica.cc:330] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: stopping tablet replica
I20260926 12:42:37.383103 28688 raft_consensus.cc:2229] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:37.383551 28688 raft_consensus.cc:2258] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:37.383951 28687 tablet_replica.cc:330] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: stopping tablet replica
I20260926 12:42:37.384253 28687 raft_consensus.cc:2229] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:37.384567 28687 raft_consensus.cc:2258] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:37.385434 28688 ts_tablet_manager.cc:1905] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:37.385874 28687 ts_tablet_manager.cc:1905] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:37.388322 28687 ts_tablet_manager.cc:1918] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 12:42:37.388374 28170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:38117 (error 108)
I20260926 12:42:37.388470 28687 log.cc:1196] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/2ed9fabdf21e461f97ab9ade3060754b
I20260926 12:42:37.388741 28688 ts_tablet_manager.cc:1918] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:37.388852 28688 log.cc:1196] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/2ed9fabdf21e461f97ab9ade3060754b
I20260926 12:42:37.388933 28687 ts_tablet_manager.cc:1939] T 2ed9fabdf21e461f97ab9ade3060754b P 0b3cc6be6dd74fbfba6d309131fbaebe: Deleting consensus metadata
I20260926 12:42:37.389171 28688 ts_tablet_manager.cc:1939] T 2ed9fabdf21e461f97ab9ade3060754b P cd106e7c31a84d42abdf3aadfce5b08e: Deleting consensus metadata
I20260926 12:42:37.389379 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 28382
I20260926 12:42:37.389868 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.389478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:38117 (local address 127.0.0.1:44482)
0926 12:42:37.389698 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.389774 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260926 12:42:37.390142 28170 catalog_manager.cc:4584] TS 9df54440aa2d4c0fa6ffe9f751d37fc4 (127.10.224.129:38117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ed9fabdf21e461f97ab9ade3060754b: Network error: Client connection negotiation failed: client connection to 127.10.224.129:38117: connect: Connection refused (error 111)
I20260926 12:42:37.390465 28172 catalog_manager.cc:4841] TS 0b3cc6be6dd74fbfba6d309131fbaebe (127.10.224.131:33739): tablet 2ed9fabdf21e461f97ab9ade3060754b (table test-table [id=e2bbf4befc3948c4b62714566e13b15f]) successfully deleted
I20260926 12:42:37.396598 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 28516
I20260926 12:42:37.397886 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.396828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:32873 (local address 127.0.0.1:47894)
0926 12:42:37.397682 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.397762 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.130:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":732}
I20260926 12:42:37.403236 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 28154
2026-09-26T12:42:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2522 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T12:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:37Z Disabled control of system clock
I20260926 12:42:37.435333 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:42461
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:42461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:42:37.554886 28693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:37.555192 28693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:37.555229 28693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:37.559810 28693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:37.559890 28693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:37.559906 28693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:37.559922 28693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:37.559937 28693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:37.565220 28693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:42461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:37.566422 28693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:37.567927 28693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:37.576998 28699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:37.577030 28698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:37.576987 28701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:37.577152 28693 server_base.cc:1034] running on GCE node
I20260926 12:42:37.577795 28693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:37.579002 28693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:37.580175 28693 hybrid_clock.cc:648] HybridClock initialized: now 1790426557580158 us; error 27 us; skew 500 ppm
I20260926 12:42:37.582238 28693 webserver.cc:460] Webserver started at http://127.10.224.190:40161/ using document root <none> and password file <none>
I20260926 12:42:37.582907 28693 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:37.582974 28693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:37.583206 28693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:37.585521 28693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "9c6ac2444261477fbeffbcf5b7725287"
format_stamp: "Formatted at 2026-09-26 12:42:37 on dist-test-slave-m3pj"
server_key: "957eab7e509396c02740959db3c194ec"
server_key_iv: "fcc8cdfb7b37a5a8ec937269fc55cb83"
server_key_version: "encryptionkey@0"
I20260926 12:42:37.586130 28693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "9c6ac2444261477fbeffbcf5b7725287"
format_stamp: "Formatted at 2026-09-26 12:42:37 on dist-test-slave-m3pj"
server_key: "957eab7e509396c02740959db3c194ec"
server_key_iv: "fcc8cdfb7b37a5a8ec937269fc55cb83"
server_key_version: "encryptionkey@0"
I20260926 12:42:37.590374 28693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:42:37.593626 28707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:37.595050 28693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:42:37.595232 28693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "9c6ac2444261477fbeffbcf5b7725287"
format_stamp: "Formatted at 2026-09-26 12:42:37 on dist-test-slave-m3pj"
server_key: "957eab7e509396c02740959db3c194ec"
server_key_iv: "fcc8cdfb7b37a5a8ec937269fc55cb83"
server_key_version: "encryptionkey@0"
I20260926 12:42:37.595378 28693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:37.613339 28693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:37.614147 28693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:37.614375 28693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:37.621929 28693 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:42461
I20260926 12:42:37.621996 28759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:42461 every 8 connection(s)
I20260926 12:42:37.623027 28693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:37.626729 28760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:37.633013 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 28693
I20260926 12:42:37.633172 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:37.633486 11138 external_mini_cluster.cc:1428] Setting key bf5481547ab9bcea0d6abfb799ebbec6
I20260926 12:42:37.633761 28760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287: Bootstrap starting.
I20260926 12:42:37.636945 28760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:37.637893 28760 log.cc:824] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:37.640434 28760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287: No bootstrap required, opened a new log
I20260926 12:42:37.644302 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.635166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.10.224.190:42461)
0926 12:42:37.636050 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.636066 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.636108 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:42:37.636845 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.636867 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.637457 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.637846 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:37.638333 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.639352 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.640365 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.640992 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.641192 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.642169 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:37.642175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:37.642192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:37.642659 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:37.642731 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:37.642755 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:37.642877 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.643010 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.643327 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":218,"threads_started":1}
I20260926 12:42:37.645123 28760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ac2444261477fbeffbcf5b7725287" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42461 } }
I20260926 12:42:37.645505 28760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:37.645577 28760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6ac2444261477fbeffbcf5b7725287, State: Initialized, Role: FOLLOWER
I20260926 12:42:37.646157 28760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ac2444261477fbeffbcf5b7725287" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42461 } }
I20260926 12:42:37.646315 28760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:37.646435 28760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:37.646574 28760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:37.648034 28760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ac2444261477fbeffbcf5b7725287" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42461 } }
I20260926 12:42:37.648514 28760 leader_election.cc:302] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c6ac2444261477fbeffbcf5b7725287; no voters:
I20260926 12:42:37.648931 28760 leader_election.cc:288] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:37.649098 28765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:37.649358 28765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6ac2444261477fbeffbcf5b7725287, State: Running, Role: LEADER
I20260926 12:42:37.649703 28765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ac2444261477fbeffbcf5b7725287" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42461 } }
I20260926 12:42:37.650514 28760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:37.651743 28766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c6ac2444261477fbeffbcf5b7725287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ac2444261477fbeffbcf5b7725287" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42461 } } }
I20260926 12:42:37.651743 28767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c6ac2444261477fbeffbcf5b7725287. Latest consensus state: current_term: 1 leader_uuid: "9c6ac2444261477fbeffbcf5b7725287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ac2444261477fbeffbcf5b7725287" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42461 } } }
I20260926 12:42:37.651916 28767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:37.651916 28766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:37.652339 28774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:37.656013 28774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:37.663110 28774 catalog_manager.cc:1347] Generated new cluster ID: 2dadc769fcc6453aa9fc896506941334
I20260926 12:42:37.663228 28774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:37.677213 28774 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:37.678251 28774 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:37.691344 28774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287: Generated new TSK 0
I20260926 12:42:37.692346 28774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:37.698944 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42461
--builtin_ntp_servers=127.10.224.148:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:42:37.816483 28784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:37.816790 28784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:37.816831 28784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:37.821094 28784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:37.821168 28784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:37.821228 28784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:42:37.826356 28784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:37.827589 28784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:37.829044 28784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:37.839104 28792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:37.839110 28789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:37.839107 28790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:37.839164 28784 server_base.cc:1034] running on GCE node
I20260926 12:42:37.840046 28784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:37.841041 28784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:37.842257 28784 hybrid_clock.cc:648] HybridClock initialized: now 1790426557842238 us; error 32 us; skew 500 ppm
I20260926 12:42:37.844452 28784 webserver.cc:460] Webserver started at http://127.10.224.129:38763/ using document root <none> and password file <none>
I20260926 12:42:37.845067 28784 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:37.845160 28784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:37.845383 28784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:37.847620 28784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "0e28b6b5a4754cec84704d50407ff29e"
format_stamp: "Formatted at 2026-09-26 12:42:37 on dist-test-slave-m3pj"
server_key: "101946feb15b26e99ace3b0f641d2a18"
server_key_iv: "1a6fb2e5770b1c6b6b0a64b353eccfa8"
server_key_version: "encryptionkey@0"
I20260926 12:42:37.848305 28784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "0e28b6b5a4754cec84704d50407ff29e"
format_stamp: "Formatted at 2026-09-26 12:42:37 on dist-test-slave-m3pj"
server_key: "101946feb15b26e99ace3b0f641d2a18"
server_key_iv: "1a6fb2e5770b1c6b6b0a64b353eccfa8"
server_key_version: "encryptionkey@0"
I20260926 12:42:37.852373 28784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 12:42:37.854923 28798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:37.856384 28784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:37.856513 28784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "0e28b6b5a4754cec84704d50407ff29e"
format_stamp: "Formatted at 2026-09-26 12:42:37 on dist-test-slave-m3pj"
server_key: "101946feb15b26e99ace3b0f641d2a18"
server_key_iv: "1a6fb2e5770b1c6b6b0a64b353eccfa8"
server_key_version: "encryptionkey@0"
I20260926 12:42:37.856627 28784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:37.877797 28784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:37.878602 28784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:37.878815 28784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:37.879467 28784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:37.880489 28784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:37.880563 28784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:37.880636 28784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:37.880679 28784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:37.889930 28784 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:33609
I20260926 12:42:37.889956 28911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:33609 every 8 connection(s)
I20260926 12:42:37.890914 28784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:42:37.896241 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 28784
I20260926 12:42:37.896369 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:42:37.896659 11138 external_mini_cluster.cc:1428] Setting key 3a336cd49b710cc3b0e411254e370032
I20260926 12:42:37.898962 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42461
--builtin_ntp_servers=127.10.224.148:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:42:37.902391 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.893023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:36655 (local address 127.10.224.190:42461)
0926 12:42:37.893186 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:37.893191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:37.894299 (+ 1108us) server_negotiation.cc:407] Connection header received
0926 12:42:37.895771 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:37.895774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:37.895824 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:37.895919 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:37.897216 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.898042 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.900064 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.900495 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.900585 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.901854 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:37.901863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:37.901865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:37.901878 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:37.901898 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:37.901906 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:37.901989 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:37.902160 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 12:42:37.902229 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:42:37.903333 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:37.893473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42461 (local address 127.10.224.129:36655)
0926 12:42:37.894152 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:37.894193 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:37.895551 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:37.896155 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:37.896165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:37.896286 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:37.897043 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.897072 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.898257 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.898261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.899900 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:37.899916 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:37.900672 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:37.900678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:37.900858 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:37.901546 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:37.901548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:37.901591 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:37.901664 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:37.901990 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:37.902002 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:37.902011 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:42:37.902160 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 12:42:37.902504 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260926 12:42:37.904796 28912 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42461
I20260926 12:42:37.905122 28912 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:37.905879 28912 heartbeater.cc:511] Master 127.10.224.190:42461 requested a full tablet report, sending...
I20260926 12:42:37.907749 28724 ts_manager.cc:194] Registered new tserver with Master: 0e28b6b5a4754cec84704d50407ff29e (127.10.224.129:33609)
I20260926 12:42:37.908926 28724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.129:36655
W20260926 12:42:38.015921 28915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:38.016196 28915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:38.016234 28915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:38.020579 28915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:38.020644 28915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:38.020751 28915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:42:38.025905 28915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:38.027096 28915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:38.028576 28915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:38.038529 28920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:38.038529 28923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:38.038861 28915 server_base.cc:1034] running on GCE node
W20260926 12:42:38.038570 28921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:38.039331 28915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:38.040287 28915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:38.041431 28915 hybrid_clock.cc:648] HybridClock initialized: now 1790426558041350 us; error 92 us; skew 500 ppm
I20260926 12:42:38.044085 28915 webserver.cc:460] Webserver started at http://127.10.224.130:33721/ using document root <none> and password file <none>
I20260926 12:42:38.044890 28915 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:38.044993 28915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:38.045252 28915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:38.048010 28915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "f9651b259a51468987c6bc4c59c69df0"
format_stamp: "Formatted at 2026-09-26 12:42:38 on dist-test-slave-m3pj"
server_key: "725d4f0d759995f4a67805be4ece7e3c"
server_key_iv: "c6e2c0cce1ef2c601bf49dfa5166c24e"
server_key_version: "encryptionkey@0"
I20260926 12:42:38.048691 28915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "f9651b259a51468987c6bc4c59c69df0"
format_stamp: "Formatted at 2026-09-26 12:42:38 on dist-test-slave-m3pj"
server_key: "725d4f0d759995f4a67805be4ece7e3c"
server_key_iv: "c6e2c0cce1ef2c601bf49dfa5166c24e"
server_key_version: "encryptionkey@0"
I20260926 12:42:38.052656 28915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:38.055116 28929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:38.056418 28915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:42:38.056548 28915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "f9651b259a51468987c6bc4c59c69df0"
format_stamp: "Formatted at 2026-09-26 12:42:38 on dist-test-slave-m3pj"
server_key: "725d4f0d759995f4a67805be4ece7e3c"
server_key_iv: "c6e2c0cce1ef2c601bf49dfa5166c24e"
server_key_version: "encryptionkey@0"
I20260926 12:42:38.056653 28915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:38.077389 28915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:38.078161 28915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:38.078372 28915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:38.079016 28915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:38.080137 28915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:38.080186 28915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:38.080257 28915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:38.080299 28915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:38.090030 28915 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:44189
I20260926 12:42:38.090051 29042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:44189 every 8 connection(s)
I20260926 12:42:38.091035 28915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:42:38.096853 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 28915
I20260926 12:42:38.097007 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:42:38.097301 11138 external_mini_cluster.cc:1428] Setting key 587765275fb3bfde8c522f9464e45416
I20260926 12:42:38.099756 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42461
--builtin_ntp_servers=127.10.224.148:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:42:38.101467 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.092968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:50369 (local address 127.10.224.190:42461)
0926 12:42:38.093105 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.093110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.094210 (+ 1100us) server_negotiation.cc:407] Connection header received
0926 12:42:38.095669 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.095673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.095737 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.095849 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:38.096873 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.097564 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.098958 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.099270 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.099408 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.100825 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:38.100828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:38.100830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:38.100843 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:38.100865 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:38.100873 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:38.100930 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.101126 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.101220 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:42:38.102362 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.093363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42461 (local address 127.10.224.130:50369)
0926 12:42:38.094051 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.094092 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:38.095396 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:38.096029 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:38.096037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:38.096122 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:38.096695 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.096716 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.097709 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.097711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.098834 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.098843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.099464 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.099470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.099728 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.100494 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:38.100498 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:38.100557 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:38.100650 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:38.100963 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:38.100975 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:38.100984 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:42:38.101123 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 12:42:38.101474 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260926 12:42:38.103641 29043 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42461
I20260926 12:42:38.103878 29043 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:38.104445 29043 heartbeater.cc:511] Master 127.10.224.190:42461 requested a full tablet report, sending...
I20260926 12:42:38.105530 28724 ts_manager.cc:194] Registered new tserver with Master: f9651b259a51468987c6bc4c59c69df0 (127.10.224.130:44189)
I20260926 12:42:38.106124 28724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.130:50369
W20260926 12:42:38.217435 29046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:38.217768 29046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:38.217839 29046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:38.222247 29046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:38.222362 29046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:38.222456 29046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:42:38.227774 29046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:38.229043 29046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:38.230581 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:38.241215 29052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:38.241215 29054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:38.241225 29051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:38.241528 29046 server_base.cc:1034] running on GCE node
I20260926 12:42:38.242039 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:38.242787 29046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:38.243986 29046 hybrid_clock.cc:648] HybridClock initialized: now 1790426558243956 us; error 34 us; skew 500 ppm
I20260926 12:42:38.245919 29046 webserver.cc:460] Webserver started at http://127.10.224.131:34327/ using document root <none> and password file <none>
I20260926 12:42:38.246572 29046 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:38.246691 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:38.246969 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:38.250218 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "14f71e2baaa2459095d3e44bb5ad55e4"
format_stamp: "Formatted at 2026-09-26 12:42:38 on dist-test-slave-m3pj"
server_key: "9129fafd6de533d0043bac6277bdb563"
server_key_iv: "e8b49594dddecf9150a903659ace8ed8"
server_key_version: "encryptionkey@0"
I20260926 12:42:38.251052 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "14f71e2baaa2459095d3e44bb5ad55e4"
format_stamp: "Formatted at 2026-09-26 12:42:38 on dist-test-slave-m3pj"
server_key: "9129fafd6de533d0043bac6277bdb563"
server_key_iv: "e8b49594dddecf9150a903659ace8ed8"
server_key_version: "encryptionkey@0"
I20260926 12:42:38.255309 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:42:38.258095 29060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:38.259433 29046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:42:38.259625 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "14f71e2baaa2459095d3e44bb5ad55e4"
format_stamp: "Formatted at 2026-09-26 12:42:38 on dist-test-slave-m3pj"
server_key: "9129fafd6de533d0043bac6277bdb563"
server_key_iv: "e8b49594dddecf9150a903659ace8ed8"
server_key_version: "encryptionkey@0"
I20260926 12:42:38.259794 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:38.293850 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:38.294724 29046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:38.294994 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:38.295796 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:38.296924 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:38.297008 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:38.297091 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:38.297149 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:38.307757 29046 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:34103
I20260926 12:42:38.307783 29173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:34103 every 8 connection(s)
I20260926 12:42:38.308842 29046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:42:38.317672 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 29046
I20260926 12:42:38.317770 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:42:38.318106 11138 external_mini_cluster.cc:1428] Setting key bb03d0d747cf19fa2e1186485d979f49
I20260926 12:42:38.319237 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.310865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:45471 (local address 127.10.224.190:42461)
0926 12:42:38.311041 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.311046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.312251 (+ 1205us) server_negotiation.cc:407] Connection header received
0926 12:42:38.313626 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.313629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.313678 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.313757 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:38.314753 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.315324 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.316625 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.316888 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.316994 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.318481 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:38.318485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:38.318488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:38.318506 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:38.318531 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:38.318542 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:38.318645 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.318938 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.319031 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 12:42:38.320273 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.311316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42461 (local address 127.10.224.131:45471)
0926 12:42:38.312083 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.312126 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:38.313413 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:38.313928 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:38.313935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:38.314024 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:38.314585 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.314598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.315548 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.315551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.316477 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.316487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.317041 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.317043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.317194 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.318065 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:38.318067 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:38.318128 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:38.318224 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:38.318681 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:38.318692 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:38.318699 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:42:38.318845 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 12:42:38.319185 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":189,"threads_started":1}
I20260926 12:42:38.321753 29174 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42461
I20260926 12:42:38.322026 29174 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:38.322645 29174 heartbeater.cc:511] Master 127.10.224.190:42461 requested a full tablet report, sending...
I20260926 12:42:38.323772 28724 ts_manager.cc:194] Registered new tserver with Master: 14f71e2baaa2459095d3e44bb5ad55e4 (127.10.224.131:34103)
I20260926 12:42:38.324394 28724 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.131:45471
I20260926 12:42:38.332976 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:42:38.339294 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.335041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.10.224.190:42461)
0926 12:42:38.335204 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.335208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.335392 (+ 184us) server_negotiation.cc:407] Connection header received
0926 12:42:38.335492 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.335494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.335608 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.335695 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:38.336009 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.336611 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.337627 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.337883 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.337978 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.338914 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:38.338918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:38.338920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:38.338935 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:38.338957 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:38.338966 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:38.339030 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.339102 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.339168 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 12:42:38.342612 28724 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:42:38.345901 28724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:42:38.364115 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.356067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.10.224.129:33609)
0926 12:42:38.357222 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.357228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.357251 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:42:38.357326 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.357401 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.357581 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.357758 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:38.359476 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.360568 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.362343 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.363264 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.363425 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.363661 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.363776 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.363902 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":141,"threads_started":1}
I20260926 12:42:38.364307 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.355843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:33609 (local address 127.0.0.1:47850)
0926 12:42:38.356683 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.356744 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:38.356906 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:38.357957 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:38.357964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:38.358007 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:38.358358 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.358364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.360691 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.360695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.362155 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.362167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.363403 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.363406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.363565 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.363619 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 12:42:38.363740 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 12:42:38.363992 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":196,"spinlock_wait_cycles":7296,"thread_start_us":110,"threads_started":1}
I20260926 12:42:38.364840 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.356712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.10.224.130:44189)
0926 12:42:38.357191 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.357204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.357235 (+ 31us) server_negotiation.cc:407] Connection header received
0926 12:42:38.357327 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.357333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.357581 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.357758 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:38.358484 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.359306 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.362640 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.363593 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.363732 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.364512 (+ 780us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.364573 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.364681 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":123,"threads_started":1}
I20260926 12:42:38.364326 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.356155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:44189 (local address 127.0.0.1:54482)
0926 12:42:38.356817 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.356832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:38.356933 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:38.357957 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:38.357964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:38.358005 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:38.358343 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.358354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.359442 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.359449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.362481 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.362490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.363863 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.363867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.363964 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.363992 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:42:38.364064 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 12:42:38.364158 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":309,"spinlock_wait_cycles":6400,"thread_start_us":105,"threads_started":1}
I20260926 12:42:38.365581 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.356736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.10.224.131:34103)
0926 12:42:38.357565 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.357582 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.357727 (+ 145us) server_negotiation.cc:407] Connection header received
0926 12:42:38.357863 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.357867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.358106 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.358277 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:38.358542 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.359402 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.362309 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.363058 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.363225 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.365225 (+ 2000us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.365298 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.365407 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":147,"threads_started":1}
I20260926 12:42:38.365859 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.356296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:34103 (local address 127.0.0.1:57994)
0926 12:42:38.357135 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.357148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:38.357228 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:38.358258 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:38.358266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:38.358284 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:38.358395 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.358402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.359564 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.359568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.362143 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.362155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.364711 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.364715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.364804 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.364826 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:42:38.364889 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 12:42:38.365737 (+ 848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":784,"thread_start_us":109,"threads_started":1}
I20260926 12:42:38.368000 28846 tablet_service.cc:1467] Processing CreateTablet for tablet e27a9dded266435b960cf22ab662344b (DEFAULT_TABLE table=test-table [id=db2a28888e404bc1aa9fe6f3603ada51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:38.368256 28975 tablet_service.cc:1467] Processing CreateTablet for tablet e27a9dded266435b960cf22ab662344b (DEFAULT_TABLE table=test-table [id=db2a28888e404bc1aa9fe6f3603ada51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:38.369158 28846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27a9dded266435b960cf22ab662344b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:38.368799 29108 tablet_service.cc:1467] Processing CreateTablet for tablet e27a9dded266435b960cf22ab662344b (DEFAULT_TABLE table=test-table [id=db2a28888e404bc1aa9fe6f3603ada51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:38.369468 28975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27a9dded266435b960cf22ab662344b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:38.369853 29108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27a9dded266435b960cf22ab662344b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:38.375813 29191 tablet_bootstrap.cc:492] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e: Bootstrap starting.
I20260926 12:42:38.377367 29193 tablet_bootstrap.cc:492] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: Bootstrap starting.
I20260926 12:42:38.379673 29191 tablet_bootstrap.cc:654] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:38.379997 29192 tablet_bootstrap.cc:492] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Bootstrap starting.
I20260926 12:42:38.380076 29193 tablet_bootstrap.cc:654] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:38.380791 29191 log.cc:824] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:38.380898 29193 log.cc:824] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:38.382186 29192 tablet_bootstrap.cc:654] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:38.382970 29192 log.cc:824] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:38.383550 29191 tablet_bootstrap.cc:492] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e: No bootstrap required, opened a new log
I20260926 12:42:38.383549 29193 tablet_bootstrap.cc:492] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: No bootstrap required, opened a new log
I20260926 12:42:38.383781 29193 ts_tablet_manager.cc:1397] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:42:38.383781 29191 ts_tablet_manager.cc:1397] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260926 12:42:38.385030 29192 tablet_bootstrap.cc:492] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: No bootstrap required, opened a new log
I20260926 12:42:38.385253 29192 ts_tablet_manager.cc:1397] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:42:38.388495 29193 raft_consensus.cc:348] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.388499 29191 raft_consensus.cc:348] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.388836 29193 raft_consensus.cc:374] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:38.388836 29191 raft_consensus.cc:374] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:38.388892 29193 raft_consensus.cc:729] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9651b259a51468987c6bc4c59c69df0, State: Initialized, Role: FOLLOWER
I20260926 12:42:38.388902 29191 raft_consensus.cc:729] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e28b6b5a4754cec84704d50407ff29e, State: Initialized, Role: FOLLOWER
I20260926 12:42:38.388902 29192 raft_consensus.cc:348] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.389205 29192 raft_consensus.cc:374] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:38.389266 29192 raft_consensus.cc:729] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f71e2baaa2459095d3e44bb5ad55e4, State: Initialized, Role: FOLLOWER
I20260926 12:42:38.389418 29193 consensus_queue.cc:260] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.389434 29191 consensus_queue.cc:260] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.389721 29192 consensus_queue.cc:260] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.390347 29043 heartbeater.cc:503] Master 127.10.224.190:42461 was elected leader, sending a full tablet report...
I20260926 12:42:38.390506 28912 heartbeater.cc:503] Master 127.10.224.190:42461 was elected leader, sending a full tablet report...
I20260926 12:42:38.390611 29191 ts_tablet_manager.cc:1428] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260926 12:42:38.391224 29193 ts_tablet_manager.cc:1428] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 12:42:38.391873 29174 heartbeater.cc:503] Master 127.10.224.190:42461 was elected leader, sending a full tablet report...
W20260926 12:42:38.392200 28913 tablet.cc:2369] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:38.392798 29192 ts_tablet_manager.cc:1428] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 12:42:38.491613 29199 raft_consensus.cc:482] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:38.491835 29199 raft_consensus.cc:504] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.492985 29199 leader_election.cc:288] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e28b6b5a4754cec84704d50407ff29e (127.10.224.129:33609), f9651b259a51468987c6bc4c59c69df0 (127.10.224.130:44189)
I20260926 12:42:38.496836 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.493309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:44189 (local address 127.10.224.131:56321)
0926 12:42:38.493429 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.493446 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:38.493559 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:38.493928 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:38.493931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:38.493975 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:38.494092 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.494098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.494828 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.494834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.495756 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.495765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.496392 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.496396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.496507 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.496525 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:42:38.496612 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:42:38.496697 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1,"spinlock_wait_cycles":3328}
I20260926 12:42:38.496836 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.493551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:56321 (local address 127.10.224.130:44189)
0926 12:42:38.493691 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.493695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.493706 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:42:38.493756 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.493758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.493798 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.493914 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:38.494215 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.494713 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.495876 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.496267 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.496399 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.496450 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.496629 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.496703 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:42:38.497010 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.493469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:55709 (local address 127.10.224.129:33609)
0926 12:42:38.493606 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.493610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.493917 (+ 307us) server_negotiation.cc:407] Connection header received
0926 12:42:38.494094 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.494098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.494150 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.494237 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:38.494506 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.494989 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.496075 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.496469 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.496601 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.496648 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.496821 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.496903 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:42:38.497027 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.493317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:33609 (local address 127.10.224.131:55709)
0926 12:42:38.493786 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.493797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:38.493980 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:38.494250 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:38.494252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:38.494268 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:38.494397 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.494403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.495099 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.495105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.495952 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:38.495961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.496591 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.496595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:38.496694 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.496717 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:42:38.496803 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 12:42:38.496868 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":45,"spinlock_wait_cycles":26752,"thread_start_us":120,"threads_started":1}
I20260926 12:42:38.497574 28866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27a9dded266435b960cf22ab662344b" candidate_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e28b6b5a4754cec84704d50407ff29e" is_pre_election: true
I20260926 12:42:38.497582 28997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27a9dded266435b960cf22ab662344b" candidate_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9651b259a51468987c6bc4c59c69df0" is_pre_election: true
I20260926 12:42:38.497889 28997 raft_consensus.cc:2454] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14f71e2baaa2459095d3e44bb5ad55e4 in term 0.
I20260926 12:42:38.497895 28866 raft_consensus.cc:2454] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14f71e2baaa2459095d3e44bb5ad55e4 in term 0.
I20260926 12:42:38.498473 29062 leader_election.cc:302] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e28b6b5a4754cec84704d50407ff29e, 14f71e2baaa2459095d3e44bb5ad55e4; no voters:
I20260926 12:42:38.498787 29199 raft_consensus.cc:2781] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:42:38.498914 29199 raft_consensus.cc:482] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:42:38.499006 29199 raft_consensus.cc:3037] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:38.500214 29199 raft_consensus.cc:504] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.500605 29199 leader_election.cc:288] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [CANDIDATE]: Term 1 election: Requested vote from peers 0e28b6b5a4754cec84704d50407ff29e (127.10.224.129:33609), f9651b259a51468987c6bc4c59c69df0 (127.10.224.130:44189)
I20260926 12:42:38.501022 28866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27a9dded266435b960cf22ab662344b" candidate_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e28b6b5a4754cec84704d50407ff29e"
I20260926 12:42:38.501063 28997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27a9dded266435b960cf22ab662344b" candidate_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9651b259a51468987c6bc4c59c69df0"
I20260926 12:42:38.501189 28866 raft_consensus.cc:3037] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:38.501226 28997 raft_consensus.cc:3037] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:38.502362 28997 raft_consensus.cc:2454] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14f71e2baaa2459095d3e44bb5ad55e4 in term 1.
I20260926 12:42:38.502458 28866 raft_consensus.cc:2454] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14f71e2baaa2459095d3e44bb5ad55e4 in term 1.
I20260926 12:42:38.502857 29064 leader_election.cc:302] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f71e2baaa2459095d3e44bb5ad55e4, f9651b259a51468987c6bc4c59c69df0; no voters:
I20260926 12:42:38.503131 29199 raft_consensus.cc:2781] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:38.503381 29199 raft_consensus.cc:686] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 1 LEADER]: Becoming Leader. State: Replica: 14f71e2baaa2459095d3e44bb5ad55e4, State: Running, Role: LEADER
I20260926 12:42:38.503798 29199 consensus_queue.cc:237] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } }
I20260926 12:42:38.507215 28724 catalog_manager.cc:5495] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14f71e2baaa2459095d3e44bb5ad55e4 (127.10.224.131). New cstate: current_term: 1 leader_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f71e2baaa2459095d3e44bb5ad55e4" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34103 } health_report { overall_health: HEALTHY } } }
I20260926 12:42:38.555063 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.550693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.10.224.131:34103)
0926 12:42:38.550833 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:38.550837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:38.550851 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:42:38.550949 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:38.550951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:38.551006 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:38.551106 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:42:38.551311 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.552057 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.553088 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:38.553299 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:38.553384 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:38.553686 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:42:38.554499 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:42:38.554647 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:38.554839 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 12:42:38.554903 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 12:42:38.559942 29175 tablet.cc:2369] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:38.562103 28866 raft_consensus.cc:1261] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [term 1 FOLLOWER]: Refusing update from remote peer 14f71e2baaa2459095d3e44bb5ad55e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:38.562951 29199 consensus_queue.cc:1035] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e28b6b5a4754cec84704d50407ff29e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:38.563076 28997 raft_consensus.cc:1261] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 1 FOLLOWER]: Refusing update from remote peer 14f71e2baaa2459095d3e44bb5ad55e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:38.563733 29199 consensus_queue.cc:1035] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9651b259a51468987c6bc4c59c69df0" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:38.570566 29204 mvcc.cc:205] Tried to move back new op lower bound from 7333587183861473280 to 7333587183634485248. Current Snapshot: MvccSnapshot[applied={T|T < 7333587183861473280}]
I20260926 12:42:38.573483 29206 mvcc.cc:205] Tried to move back new op lower bound from 7333587183861473280 to 7333587183634485248. Current Snapshot: MvccSnapshot[applied={T|T < 7333587183861473280}]
I20260926 12:42:38.580961 28724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:38.581127 28724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:38.583803 28724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c6ac2444261477fbeffbcf5b7725287: Sending DeleteTablet for 3 replicas of tablet e27a9dded266435b960cf22ab662344b
I20260926 12:42:38.584862 28846 tablet_service.cc:1514] Processing DeleteTablet for tablet e27a9dded266435b960cf22ab662344b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:38 UTC) from {username='slave'} at 127.0.0.1:47850
I20260926 12:42:38.584849 28975 tablet_service.cc:1514] Processing DeleteTablet for tablet e27a9dded266435b960cf22ab662344b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:38 UTC) from {username='slave'} at 127.0.0.1:54482
I20260926 12:42:38.585183 29108 tablet_service.cc:1514] Processing DeleteTablet for tablet e27a9dded266435b960cf22ab662344b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:38 UTC) from {username='slave'} at 127.0.0.1:57994
I20260926 12:42:38.585742 29213 tablet_replica.cc:330] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e: stopping tablet replica
I20260926 12:42:38.585810 29215 tablet_replica.cc:330] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: stopping tablet replica
I20260926 12:42:38.585892 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 28784
I20260926 12:42:38.586170 29213 raft_consensus.cc:2229] T e27a9dded266435b960cf22ab662344b P 0e28b6b5a4754cec84704d50407ff29e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:38.586186 29215 raft_consensus.cc:2229] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:42:38.586668 29215 raft_consensus.cc:2258] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:38.587481 29214 tablet_replica.cc:330] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: stopping tablet replica
I20260926 12:42:38.587870 29214 raft_consensus.cc:2229] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:38.588238 29214 raft_consensus.cc:2258] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:38.589105 29215 ts_tablet_manager.cc:1905] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:38.591702 29215 ts_tablet_manager.cc:1918] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:38.591814 29215 log.cc:1196] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/e27a9dded266435b960cf22ab662344b
I20260926 12:42:38.592127 29215 ts_tablet_manager.cc:1939] T e27a9dded266435b960cf22ab662344b P 14f71e2baaa2459095d3e44bb5ad55e4: Deleting consensus metadata
I20260926 12:42:38.593144 29214 ts_tablet_manager.cc:1905] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:38.593773 28710 catalog_manager.cc:4841] TS 14f71e2baaa2459095d3e44bb5ad55e4 (127.10.224.131:34103): tablet e27a9dded266435b960cf22ab662344b (table test-table [id=db2a28888e404bc1aa9fe6f3603ada51]) successfully deleted
I20260926 12:42:38.595484 29214 ts_tablet_manager.cc:1918] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:38.595664 29214 log.cc:1196] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/e27a9dded266435b960cf22ab662344b
I20260926 12:42:38.596016 29214 ts_tablet_manager.cc:1939] T e27a9dded266435b960cf22ab662344b P f9651b259a51468987c6bc4c59c69df0: Deleting consensus metadata
W20260926 12:42:38.596127 28709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:33609 (error 108)
I20260926 12:42:38.596354 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 28915
I20260926 12:42:38.597982 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.597624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:33609 (local address 127.0.0.1:47856)
0926 12:42:38.597813 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.597890 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:33609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260926 12:42:38.598271 28709 catalog_manager.cc:4584] TS 0e28b6b5a4754cec84704d50407ff29e (127.10.224.129:33609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e27a9dded266435b960cf22ab662344b: Network error: Client connection negotiation failed: client connection to 127.10.224.129:33609: connect: Connection refused (error 111)
I20260926 12:42:38.603271 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 29046
I20260926 12:42:38.603513 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:38.603168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:44189 (local address 127.0.0.1:54494)
0926 12:42:38.603330 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:38.603401 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.130:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 12:42:38.608999 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 28693
2026-09-26T12:42:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1204 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:38 dist-test-slave-m3pj krb5kdc[29219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:38 dist-test-slave-m3pj krb5kdc[29219](info): set up 2 sockets
Sep 26 12:42:38 dist-test-slave-m3pj krb5kdc[29219](info): commencing operation
krb5kdc: starting...
W20260926 12:42:39.659756 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:39 dist-test-slave-m3pj krb5kdc[29219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:42:39.788028 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:33927
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:33927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:39.901089 29232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:39.901394 29232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:39.901444 29232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:39.905826 29232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:39.905900 29232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:39.905923 29232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:39.905941 29232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:39.905957 29232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:39.911610 29232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:33927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:33927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:39.912842 29232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:39.913936 29232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:39.923110 29240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:39.923130 29238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:39.923110 29237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:39.923382 29232 server_base.cc:1034] running on GCE node
I20260926 12:42:39.924014 29232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:39.925200 29232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:39.926396 29232 hybrid_clock.cc:648] HybridClock initialized: now 1790426559926378 us; error 31 us; skew 500 ppm
Sep 26 12:42:39 dist-test-slave-m3pj krb5kdc[29219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426559, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:39.929311 29232 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:42:39.930310 29232 webserver.cc:460] Webserver started at http://127.10.224.190:35737/ using document root <none> and password file <none>
I20260926 12:42:39.930948 29232 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:39.931046 29232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:39.931293 29232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:39.933557 29232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "0576059902ad4bd0a49604ff375b67a0"
format_stamp: "Formatted at 2026-09-26 12:42:39 on dist-test-slave-m3pj"
server_key: "7d11592a783478715a76adc575ac4be9"
server_key_iv: "60f27d1b9d28161622d538dbc04a4cf1"
server_key_version: "encryptionkey@0"
I20260926 12:42:39.934206 29232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "0576059902ad4bd0a49604ff375b67a0"
format_stamp: "Formatted at 2026-09-26 12:42:39 on dist-test-slave-m3pj"
server_key: "7d11592a783478715a76adc575ac4be9"
server_key_iv: "60f27d1b9d28161622d538dbc04a4cf1"
server_key_version: "encryptionkey@0"
I20260926 12:42:39.938123 29232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:42:39.940762 29247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:39.942394 29232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:42:39.942554 29232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "0576059902ad4bd0a49604ff375b67a0"
format_stamp: "Formatted at 2026-09-26 12:42:39 on dist-test-slave-m3pj"
server_key: "7d11592a783478715a76adc575ac4be9"
server_key_iv: "60f27d1b9d28161622d538dbc04a4cf1"
server_key_version: "encryptionkey@0"
I20260926 12:42:39.942701 29232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:39.959981 29232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:39.963445 29232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:39.963706 29232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:39.971131 29232 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:33927
I20260926 12:42:39.971151 29299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:33927 every 8 connection(s)
I20260926 12:42:39.972225 29232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:39.974517 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 29232
I20260926 12:42:39.974673 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:39.974978 11138 external_mini_cluster.cc:1428] Setting key 573b7300521e525b705c87ef5f8661c3
I20260926 12:42:39.975329 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:39.982036 29300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0: Bootstrap starting.
Sep 26 12:42:39 dist-test-slave-m3pj krb5kdc[29219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:39.984982 29300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:39.985958 29300 log.cc:824] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:39.988229 29300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0: No bootstrap required, opened a new log
I20260926 12:42:39.990302 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:39.976640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.10.224.190:33927)
0926 12:42:39.977144 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:39.977160 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:39.977196 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:42:39.977863 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:39.977880 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:39.978222 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:39.978678 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:39.979569 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:39.980591 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:39.981464 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:39.981962 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:39.982145 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:39.985131 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:39.985138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:39.985160 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:39.985200 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:39.987268 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:39.987857 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:39.987863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:39.987867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:39.987988 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:39.988279 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:39.988284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:39.988285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:39.988754 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:39.988973 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:39.989237 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 12:42:39.989552 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":183,"threads_started":1}
I20260926 12:42:39.992064 29300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0576059902ad4bd0a49604ff375b67a0" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33927 } }
I20260926 12:42:39.992314 29300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:39.992398 29300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0576059902ad4bd0a49604ff375b67a0, State: Initialized, Role: FOLLOWER
I20260926 12:42:39.992857 29300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0576059902ad4bd0a49604ff375b67a0" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33927 } }
I20260926 12:42:39.993016 29300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:39.993079 29300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:39.993206 29300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:39.994177 29300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0576059902ad4bd0a49604ff375b67a0" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33927 } }
I20260926 12:42:39.994571 29300 leader_election.cc:302] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0576059902ad4bd0a49604ff375b67a0; no voters:
I20260926 12:42:39.994892 29300 leader_election.cc:288] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:39.995074 29305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:39.995324 29305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [term 1 LEADER]: Becoming Leader. State: Replica: 0576059902ad4bd0a49604ff375b67a0, State: Running, Role: LEADER
I20260926 12:42:39.995721 29305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0576059902ad4bd0a49604ff375b67a0" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33927 } }
I20260926 12:42:39.996268 29300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:39.997452 29307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0576059902ad4bd0a49604ff375b67a0. Latest consensus state: current_term: 1 leader_uuid: "0576059902ad4bd0a49604ff375b67a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0576059902ad4bd0a49604ff375b67a0" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33927 } } }
I20260926 12:42:39.997594 29307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:39.997941 29306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0576059902ad4bd0a49604ff375b67a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0576059902ad4bd0a49604ff375b67a0" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 33927 } } }
I20260926 12:42:39.998037 29306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:39.998237 29314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:40.001410 29314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:40.007170 29314 catalog_manager.cc:1347] Generated new cluster ID: ce5e2b57add64e60bd9a37e261695723
I20260926 12:42:40.007259 29314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:40.015084 29314 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:40.016440 29314 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:40.025453 29314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0: Generated new TSK 0
I20260926 12:42:40.026238 29314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:42:40.058951 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33927
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:40.175990 29326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:40.176250 29326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:40.176296 29326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:40.180405 29326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:40.180471 29326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:40.180583 29326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:42:40.186636 29326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:40.187829 29326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:40.188880 29326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:40.198902 29332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:40.198889 29334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:40.198889 29331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:40.199151 29326 server_base.cc:1034] running on GCE node
I20260926 12:42:40.199697 29326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:40.200353 29326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:40.201530 29326 hybrid_clock.cc:648] HybridClock initialized: now 1790426560201510 us; error 31 us; skew 500 ppm
Sep 26 12:42:40 dist-test-slave-m3pj krb5kdc[29219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:40.204550 29326 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:42:40.205497 29326 webserver.cc:460] Webserver started at http://127.10.224.129:42531/ using document root <none> and password file <none>
I20260926 12:42:40.206172 29326 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:40.206267 29326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:40.206496 29326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:40.208818 29326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "29f1fd34068a4f269669312438aa2671"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "07dfed3f174e7345ba1da2ba28cd6ddf"
server_key_iv: "abce32c8b769eef6ee8e7bae38bbd7d9"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.209436 29326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "29f1fd34068a4f269669312438aa2671"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "07dfed3f174e7345ba1da2ba28cd6ddf"
server_key_iv: "abce32c8b769eef6ee8e7bae38bbd7d9"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.213528 29326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:42:40.216400 29341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.217741 29326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:40.217890 29326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "29f1fd34068a4f269669312438aa2671"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "07dfed3f174e7345ba1da2ba28cd6ddf"
server_key_iv: "abce32c8b769eef6ee8e7bae38bbd7d9"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.218021 29326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:40.250532 29326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:40.254177 29326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:40.254405 29326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:40.255053 29326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:40.255975 29326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:40.256047 29326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.256119 29326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:40.256170 29326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.265674 29326 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:42451
I20260926 12:42:40.265826 29454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:42451 every 8 connection(s)
I20260926 12:42:40.266875 29326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:42:40.276849 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 29326
I20260926 12:42:40.277029 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:42:40.277347 11138 external_mini_cluster.cc:1428] Setting key 2df5c7153d64596f9037889002e747f5
Sep 26 12:42:40 dist-test-slave-m3pj krb5kdc[29219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:40.282650 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.268933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:36821 (local address 127.10.224.190:33927)
0926 12:42:40.269089 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:40.269093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:40.270050 (+ 957us) server_negotiation.cc:407] Connection header received
0926 12:42:40.271407 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:40.271411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:40.271475 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:40.271630 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:40.272853 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.273464 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.274902 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.275205 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.275319 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.278289 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:40.278293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:40.278295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:40.278318 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:40.280142 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:40.280813 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:40.280818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:40.280821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:40.280929 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:40.281374 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:40.281376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:40.281378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:40.281569 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:40.281647 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:40.282401 (+ 754us) server_negotiation.cc:299] Negotiation successful
0926 12:42:40.282521 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:42:40.284081 29457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.269422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33927 (local address 127.10.224.129:36821)
0926 12:42:40.269881 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:40.269926 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:40.271126 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:40.271770 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:40.271777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:40.272147 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:40.272685 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.272700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.273624 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.273627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.274744 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.274756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.275409 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.275412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.275659 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.276322 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:40.276339 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:40.278134 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:40.280288 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:40.280294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:40.280310 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:40.280670 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:40.281038 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:40.281042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:40.281047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:40.281270 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:40.281676 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:40.281684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:40.282100 (+ 416us) client_negotiation.cc:769] Sending connection context
0926 12:42:40.282349 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 12:42:40.282763 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260926 12:42:40.285720 29455 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33927
I20260926 12:42:40.286010 29455 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:40.286540 29455 heartbeater.cc:511] Master 127.10.224.190:33927 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:42:40.288350 29264 ts_manager.cc:194] Registered new tserver with Master: 29f1fd34068a4f269669312438aa2671 (127.10.224.129:42451)
I20260926 12:42:40.289616 29264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:36821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:42:40.311146 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33927
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:40.426199 29460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:40.426482 29460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:40.426527 29460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:40.430562 29460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:40.430631 29460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:40.430691 29460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:42:40.436106 29460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:40.437198 29460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:40.438235 29460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:40.448994 29468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:40.449069 29460 server_base.cc:1034] running on GCE node
W20260926 12:42:40.448998 29465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:40.449055 29466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:40.449800 29460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:40.450562 29460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:40.451749 29460 hybrid_clock.cc:648] HybridClock initialized: now 1790426560451728 us; error 35 us; skew 500 ppm
Sep 26 12:42:40 dist-test-slave-m3pj krb5kdc[29219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:40.454664 29460 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:42:40.455765 29460 webserver.cc:460] Webserver started at http://127.10.224.130:44139/ using document root <none> and password file <none>
I20260926 12:42:40.456396 29460 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:40.456498 29460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:40.456741 29460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:40.458938 29460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "0e68ff41c57d4d598a12d4ec300a3928"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "cbb75649d55c747ef1996d5da9756cdb"
server_key_iv: "0f882fcec5bab3ecd80eed66105457e1"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.459604 29460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "0e68ff41c57d4d598a12d4ec300a3928"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "cbb75649d55c747ef1996d5da9756cdb"
server_key_iv: "0f882fcec5bab3ecd80eed66105457e1"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.463657 29460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:40.466183 29475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.467602 29460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:40.467736 29460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "0e68ff41c57d4d598a12d4ec300a3928"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "cbb75649d55c747ef1996d5da9756cdb"
server_key_iv: "0f882fcec5bab3ecd80eed66105457e1"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.467851 29460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:40.486061 29460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:40.489446 29460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:40.489689 29460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:40.490300 29460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:40.491184 29460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:40.491257 29460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.491328 29460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:40.491380 29460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.501142 29460 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:35073
I20260926 12:42:40.501163 29588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:35073 every 8 connection(s)
I20260926 12:42:40.502056 29460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:42:40.508450 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 29460
I20260926 12:42:40.508587 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:42:40.508893 11138 external_mini_cluster.cc:1428] Setting key e19d7c63ff765e54dbb34777835f46f1
Sep 26 12:42:40 dist-test-slave-m3pj krb5kdc[29219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:40.516672 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.503954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:56149 (local address 127.10.224.190:33927)
0926 12:42:40.504090 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:40.504094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:40.504944 (+ 850us) server_negotiation.cc:407] Connection header received
0926 12:42:40.506272 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:40.506275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:40.506324 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:40.506419 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:40.507762 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.508362 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.509791 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.510137 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.510258 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.512990 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:40.512994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:40.512996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:40.513020 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:40.514574 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:40.515099 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:40.515104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:40.515106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:40.515188 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:40.515551 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:40.515556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:40.515558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:40.515742 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:40.515851 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:40.516374 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 12:42:40.516507 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:42:40.517549 29591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.504360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33927 (local address 127.10.224.130:56149)
0926 12:42:40.504782 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:40.504825 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:40.506077 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:40.506664 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:40.506671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:40.507038 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:40.507620 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.507636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.508491 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.508493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.509683 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.509693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.510298 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.510302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.510486 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.511120 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:40.511136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:40.512808 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:40.514692 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:40.514698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:40.514715 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:40.514992 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:40.515281 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:40.515285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:40.515287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:40.515438 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:40.515850 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:40.515857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:40.516115 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 12:42:40.516323 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 12:42:40.516678 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
I20260926 12:42:40.518810 29589 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33927
I20260926 12:42:40.519049 29589 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:40.519570 29589 heartbeater.cc:511] Master 127.10.224.190:33927 requested a full tablet report, sending...
I20260926 12:42:40.520685 29264 ts_manager.cc:194] Registered new tserver with Master: 0e68ff41c57d4d598a12d4ec300a3928 (127.10.224.130:35073)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:42:40.521333 29264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:56149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:42:40.539199 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33927
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:40.653973 29594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:40.654255 29594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:40.654330 29594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:40.658537 29594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:40.658622 29594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:40.658718 29594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:42:40.664590 29594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:40.665812 29594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:40.666906 29594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:40.676894 29600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:40.676985 29594 server_base.cc:1034] running on GCE node
W20260926 12:42:40.676909 29599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:40.676915 29602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:40.677647 29594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:40.678297 29594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:40.679487 29594 hybrid_clock.cc:648] HybridClock initialized: now 1790426560679460 us; error 36 us; skew 500 ppm
Sep 26 12:42:40 dist-test-slave-m3pj krb5kdc[29219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:40.682519 29594 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:42:40.683444 29594 webserver.cc:460] Webserver started at http://127.10.224.131:37503/ using document root <none> and password file <none>
I20260926 12:42:40.684056 29594 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:40.684150 29594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:40.684387 29594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:40.686491 29594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "1b1bc94a9e4d4c9b9987a67f97edac05"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "9d910d07d748797d55db083499ef4eb4"
server_key_iv: "ff0d09b6f6ad078e3d80d7da6a32f3c5"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.687059 29594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "1b1bc94a9e4d4c9b9987a67f97edac05"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "9d910d07d748797d55db083499ef4eb4"
server_key_iv: "ff0d09b6f6ad078e3d80d7da6a32f3c5"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.690977 29594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:42:40.693574 29609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.694805 29594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:42:40.694965 29594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "1b1bc94a9e4d4c9b9987a67f97edac05"
format_stamp: "Formatted at 2026-09-26 12:42:40 on dist-test-slave-m3pj"
server_key: "9d910d07d748797d55db083499ef4eb4"
server_key_iv: "ff0d09b6f6ad078e3d80d7da6a32f3c5"
server_key_version: "encryptionkey@0"
I20260926 12:42:40.695089 29594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:40.709314 29594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:40.712446 29594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:40.712661 29594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:40.713271 29594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:40.714134 29594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:40.714206 29594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.714277 29594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:40.714329 29594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:40.724195 29594 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:34661
I20260926 12:42:40.724238 29722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:34661 every 8 connection(s)
I20260926 12:42:40.725142 29594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:42:40.726024 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 29594
I20260926 12:42:40.726174 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:42:40.726541 11138 external_mini_cluster.cc:1428] Setting key b7bb272dfd6253577ff1221eb3c5649e
Sep 26 12:42:40 dist-test-slave-m3pj krb5kdc[29219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:40.740218 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.727452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:36235 (local address 127.10.224.190:33927)
0926 12:42:40.727647 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:40.727652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:40.728853 (+ 1201us) server_negotiation.cc:407] Connection header received
0926 12:42:40.730445 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:40.730448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:40.730495 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:40.730562 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:40.731967 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.732508 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.733672 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.733893 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.733968 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.736561 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:40.736565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:40.736567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:40.736594 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:40.737960 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:40.738489 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:40.738491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:40.738493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:40.738585 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:40.738893 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:40.738895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:40.738896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:40.739127 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:40.739222 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:40.739860 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 12:42:40.740024 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 12:42:40.741129 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.728151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:33927 (local address 127.10.224.131:36235)
0926 12:42:40.728679 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:40.728740 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:40.730223 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:40.730712 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:40.730719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:40.731112 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:40.731808 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.731844 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.732644 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.732647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.733561 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.733571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.734034 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.734037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.734180 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.734738 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:40.734755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:40.736375 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:40.738077 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:40.738082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:40.738098 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:40.738389 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:40.738661 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:40.738663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:40.738665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:40.738808 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:40.739221 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:40.739226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:40.739577 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 12:42:40.739766 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 12:42:40.740185 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":154,"threads_started":1}
I20260926 12:42:40.742319 29723 heartbeater.cc:347] Connected to a master server at 127.10.224.190:33927
I20260926 12:42:40.742571 29723 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:40.743016 29723 heartbeater.cc:511] Master 127.10.224.190:33927 requested a full tablet report, sending...
I20260926 12:42:40.744093 29264 ts_manager.cc:194] Registered new tserver with Master: 1b1bc94a9e4d4c9b9987a67f97edac05 (127.10.224.131:34661)
I20260926 12:42:40.744632 29264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:36235
I20260926 12:42:40.752876 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:42:40.763310 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.754933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.10.224.190:33927)
0926 12:42:40.755109 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:40.755114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:40.755265 (+ 151us) server_negotiation.cc:407] Connection header received
0926 12:42:40.755463 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:40.755466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:40.755509 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:40.755651 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:40.756436 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.756967 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.757842 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.758070 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.758185 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.759844 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:40.759848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:40.759851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:40.759885 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:40.761292 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:40.761880 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:40.761884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:40.761886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:40.761973 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:40.762326 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:40.762329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:40.762331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:40.762573 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:40.762739 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:40.763030 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 12:42:40.763132 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 12:42:40.766844 29264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:42:40.770010 29264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:42:40.785390 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.778902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:35073 (local address 127.0.0.1:49358)
0926 12:42:40.779324 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:40.779377 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:40.779507 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:40.780243 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:40.780249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:40.780273 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:40.780500 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.780513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.782145 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.782148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.783603 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.783613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.784980 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.784984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.785096 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.785122 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:42:40.785173 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 12:42:40.785242 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":167,"thread_start_us":100,"threads_started":1}
I20260926 12:42:40.785398 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.779673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.10.224.131:34661)
0926 12:42:40.780107 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:40.780119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:40.780140 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:42:40.780216 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:40.780222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:40.780420 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:40.780578 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:40.780812 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.782003 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.783830 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.784639 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.784777 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.784932 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:40.785144 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 12:42:40.785251 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":165,"threads_started":1}
I20260926 12:42:40.785398 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.778960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:34661 (local address 127.0.0.1:49258)
0926 12:42:40.779766 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:40.779778 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:40.779864 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:40.780551 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:40.780555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:40.780577 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:40.780680 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.780686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.782194 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.782196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.783703 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.783714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.784787 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.784791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.784929 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.784974 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 12:42:40.785091 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 12:42:40.785191 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":108,"thread_start_us":87,"threads_started":1}
I20260926 12:42:40.785727 29737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.779667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42451 (local address 127.0.0.1:47456)
0926 12:42:40.780830 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:40.780844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:40.780975 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:40.781382 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:40.781385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:40.781402 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:40.781504 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.781512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.782686 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.782690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.783947 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:40.783954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.785326 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.785329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:40.785422 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.785445 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:42:40.785520 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 12:42:40.785593 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":152,"thread_start_us":89,"threads_started":1}
I20260926 12:42:40.786128 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.780077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.10.224.129:42451)
0926 12:42:40.780465 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:40.780480 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:40.780953 (+ 473us) server_negotiation.cc:407] Connection header received
0926 12:42:40.781116 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:40.781120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:40.781246 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:40.781390 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:40.781622 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.782528 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.784056 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.784647 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.784806 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.784951 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:40.785883 (+ 932us) server_negotiation.cc:299] Negotiation successful
0926 12:42:40.785990 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":105,"threads_started":1}
I20260926 12:42:40.786459 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:40.778969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.10.224.130:35073)
0926 12:42:40.779711 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:40.779726 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:40.779753 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:42:40.779836 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:40.779841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:40.780054 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:40.780663 (+ 609us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:40.780817 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.782003 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.783739 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:40.784837 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:40.786036 (+ 1199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:40.786165 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:40.786213 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 12:42:40.786303 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":126,"threads_started":1}
I20260926 12:42:40.788954 29523 tablet_service.cc:1467] Processing CreateTablet for tablet 750a8e8452f641d9b3db92e98d7182e5 (DEFAULT_TABLE table=test-table [id=14efc5ce07b84dce92ea4bbe91241522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:40.788950 29389 tablet_service.cc:1467] Processing CreateTablet for tablet 750a8e8452f641d9b3db92e98d7182e5 (DEFAULT_TABLE table=test-table [id=14efc5ce07b84dce92ea4bbe91241522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:40.788954 29657 tablet_service.cc:1467] Processing CreateTablet for tablet 750a8e8452f641d9b3db92e98d7182e5 (DEFAULT_TABLE table=test-table [id=14efc5ce07b84dce92ea4bbe91241522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:40.789891 29523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750a8e8452f641d9b3db92e98d7182e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:40.789964 29389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750a8e8452f641d9b3db92e98d7182e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:40.789964 29657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750a8e8452f641d9b3db92e98d7182e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:40.797652 29742 tablet_bootstrap.cc:492] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Bootstrap starting.
I20260926 12:42:40.797775 29741 tablet_bootstrap.cc:492] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671: Bootstrap starting.
I20260926 12:42:40.799925 29741 tablet_bootstrap.cc:654] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:40.799942 29742 tablet_bootstrap.cc:654] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:40.799911 29740 tablet_bootstrap.cc:492] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Bootstrap starting.
I20260926 12:42:40.800742 29741 log.cc:824] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:40.801256 29742 log.cc:824] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:40.802220 29740 tablet_bootstrap.cc:654] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:40.803112 29741 tablet_bootstrap.cc:492] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671: No bootstrap required, opened a new log
I20260926 12:42:40.803337 29741 ts_tablet_manager.cc:1397] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:42:40.803573 29740 log.cc:824] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:40.803721 29742 tablet_bootstrap.cc:492] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: No bootstrap required, opened a new log
I20260926 12:42:40.803901 29742 ts_tablet_manager.cc:1397] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:42:40.805743 29740 tablet_bootstrap.cc:492] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: No bootstrap required, opened a new log
I20260926 12:42:40.805945 29740 ts_tablet_manager.cc:1397] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:42:40.807416 29741 raft_consensus.cc:348] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:40.807757 29741 raft_consensus.cc:374] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:40.807814 29741 raft_consensus.cc:729] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f1fd34068a4f269669312438aa2671, State: Initialized, Role: FOLLOWER
I20260926 12:42:40.807792 29742 raft_consensus.cc:348] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:40.808048 29742 raft_consensus.cc:374] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:40.808097 29742 raft_consensus.cc:729] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1bc94a9e4d4c9b9987a67f97edac05, State: Initialized, Role: FOLLOWER
I20260926 12:42:40.808384 29741 consensus_queue.cc:260] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:40.808532 29742 consensus_queue.cc:260] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:40.809276 29723 heartbeater.cc:503] Master 127.10.224.190:33927 was elected leader, sending a full tablet report...
I20260926 12:42:40.809518 29741 ts_tablet_manager.cc:1428] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 12:42:40.809836 29455 heartbeater.cc:503] Master 127.10.224.190:33927 was elected leader, sending a full tablet report...
I20260926 12:42:40.809854 29740 raft_consensus.cc:348] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:40.810115 29740 raft_consensus.cc:374] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:40.810168 29740 raft_consensus.cc:729] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e68ff41c57d4d598a12d4ec300a3928, State: Initialized, Role: FOLLOWER
I20260926 12:42:40.810572 29740 consensus_queue.cc:260] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:40.811244 29589 heartbeater.cc:503] Master 127.10.224.190:33927 was elected leader, sending a full tablet report...
I20260926 12:42:40.811595 29740 ts_tablet_manager.cc:1428] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:42:40.812916 29742 ts_tablet_manager.cc:1428] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260926 12:42:40.976154 29724 tablet.cc:2369] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:40.998203 29746 raft_consensus.cc:482] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:40.998476 29746 raft_consensus.cc:504] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:40.999812 29746 leader_election.cc:288] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e68ff41c57d4d598a12d4ec300a3928 (127.10.224.130:35073), 1b1bc94a9e4d4c9b9987a67f97edac05 (127.10.224.131:34661)
W20260926 12:42:41.003448 29590 tablet.cc:2369] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:41.005605 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.000264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:38801 (local address 127.10.224.130:35073)
0926 12:42:41.000404 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:41.000410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:41.000758 (+ 348us) server_negotiation.cc:407] Connection header received
0926 12:42:41.000941 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:41.000944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:41.001003 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:41.001095 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:41.001542 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.002354 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.004267 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.004877 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.005044 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.005099 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:41.005384 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 12:42:41.005467 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:42:41.005605 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.000263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:42381 (local address 127.10.224.131:34661)
0926 12:42:41.000404 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:41.000410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:41.000758 (+ 348us) server_negotiation.cc:407] Connection header received
0926 12:42:41.000941 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:41.000944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:41.001003 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:41.001093 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:41.001542 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.002349 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.004268 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.004924 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.005044 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.005099 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:41.005414 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 12:42:41.005484 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:42:41.005787 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.000188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:34661 (local address 127.10.224.129:42381)
0926 12:42:41.000620 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:41.000636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:41.000799 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:41.001124 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:41.001127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:41.001154 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:41.001392 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:41.001399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.002489 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.002492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:41.004121 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:41.004135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.005046 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.005050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:41.005212 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.005271 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 12:42:41.005353 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 12:42:41.005568 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":361,"spinlock_wait_cycles":29312,"thread_start_us":133,"threads_started":1}
I20260926 12:42:41.005787 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.000188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:35073 (local address 127.10.224.129:38801)
0926 12:42:41.000645 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:41.000658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:41.000799 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:41.001124 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:41.001127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:41.001155 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:41.001392 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:41.001399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.002489 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.002492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:41.004121 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:41.004135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.005046 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.005050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:41.005208 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.005274 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 12:42:41.005353 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:42:41.005567 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":289,"spinlock_wait_cycles":18560,"thread_start_us":94,"threads_started":1}
I20260926 12:42:41.006671 29677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750a8e8452f641d9b3db92e98d7182e5" candidate_uuid: "29f1fd34068a4f269669312438aa2671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" is_pre_election: true
I20260926 12:42:41.006668 29543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750a8e8452f641d9b3db92e98d7182e5" candidate_uuid: "29f1fd34068a4f269669312438aa2671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e68ff41c57d4d598a12d4ec300a3928" is_pre_election: true
I20260926 12:42:41.006983 29677 raft_consensus.cc:2454] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f1fd34068a4f269669312438aa2671 in term 0.
I20260926 12:42:41.006989 29543 raft_consensus.cc:2454] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f1fd34068a4f269669312438aa2671 in term 0.
I20260926 12:42:41.007532 29345 leader_election.cc:302] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e68ff41c57d4d598a12d4ec300a3928, 29f1fd34068a4f269669312438aa2671; no voters:
I20260926 12:42:41.007865 29746 raft_consensus.cc:2781] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:42:41.007949 29746 raft_consensus.cc:482] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:42:41.007982 29746 raft_consensus.cc:3037] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:41.009099 29746 raft_consensus.cc:504] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:41.009531 29746 leader_election.cc:288] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [CANDIDATE]: Term 1 election: Requested vote from peers 0e68ff41c57d4d598a12d4ec300a3928 (127.10.224.130:35073), 1b1bc94a9e4d4c9b9987a67f97edac05 (127.10.224.131:34661)
I20260926 12:42:41.009930 29543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750a8e8452f641d9b3db92e98d7182e5" candidate_uuid: "29f1fd34068a4f269669312438aa2671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e68ff41c57d4d598a12d4ec300a3928"
I20260926 12:42:41.009941 29677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750a8e8452f641d9b3db92e98d7182e5" candidate_uuid: "29f1fd34068a4f269669312438aa2671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05"
I20260926 12:42:41.010083 29677 raft_consensus.cc:3037] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:41.010054 29543 raft_consensus.cc:3037] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:41.011183 29677 raft_consensus.cc:2454] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f1fd34068a4f269669312438aa2671 in term 1.
I20260926 12:42:41.011210 29543 raft_consensus.cc:2454] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f1fd34068a4f269669312438aa2671 in term 1.
I20260926 12:42:41.011584 29345 leader_election.cc:302] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e68ff41c57d4d598a12d4ec300a3928, 29f1fd34068a4f269669312438aa2671; no voters:
I20260926 12:42:41.011746 29746 raft_consensus.cc:2781] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:41.012017 29746 raft_consensus.cc:686] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [term 1 LEADER]: Becoming Leader. State: Replica: 29f1fd34068a4f269669312438aa2671, State: Running, Role: LEADER
I20260926 12:42:41.012403 29746 consensus_queue.cc:237] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } }
I20260926 12:42:41.016031 29263 catalog_manager.cc:5495] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f1fd34068a4f269669312438aa2671 (127.10.224.129). New cstate: current_term: 1 leader_uuid: "29f1fd34068a4f269669312438aa2671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f1fd34068a4f269669312438aa2671" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 42451 } health_report { overall_health: HEALTHY } } }
W20260926 12:42:41.018325 29456 tablet.cc:2369] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:41.081177 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.077497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.10.224.129:42451)
0926 12:42:41.077633 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:41.077637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:41.077654 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:42:41.077750 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:41.077752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:41.077809 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:41.077899 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:42:41.078164 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.078680 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.079624 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.079838 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.079921 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.080278 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:42:41.080714 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:42:41.080835 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:41.081012 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 12:42:41.081054 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:42:41.087927 29677 raft_consensus.cc:1261] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 1 FOLLOWER]: Refusing update from remote peer 29f1fd34068a4f269669312438aa2671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:41.087926 29543 raft_consensus.cc:1261] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 1 FOLLOWER]: Refusing update from remote peer 29f1fd34068a4f269669312438aa2671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:41.088724 29751 consensus_queue.cc:1035] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e68ff41c57d4d598a12d4ec300a3928" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:41.089003 29746 consensus_queue.cc:1035] T 750a8e8452f641d9b3db92e98d7182e5 P 29f1fd34068a4f269669312438aa2671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1bc94a9e4d4c9b9987a67f97edac05" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:41.099210 29755 mvcc.cc:205] Tried to move back new op lower bound from 7333587194207522816 to 7333587193910013952. Current Snapshot: MvccSnapshot[applied={T|T < 7333587194207522816}]
I20260926 12:42:41.107782 29264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:41.108017 29264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:41.110647 29264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0576059902ad4bd0a49604ff375b67a0: Sending DeleteTablet for 3 replicas of tablet 750a8e8452f641d9b3db92e98d7182e5
I20260926 12:42:41.111639 29389 tablet_service.cc:1514] Processing DeleteTablet for tablet 750a8e8452f641d9b3db92e98d7182e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:41 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:47456
I20260926 12:42:41.111626 29657 tablet_service.cc:1514] Processing DeleteTablet for tablet 750a8e8452f641d9b3db92e98d7182e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:41 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:49258
I20260926 12:42:41.112239 29763 tablet_replica.cc:330] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: stopping tablet replica
I20260926 12:42:41.112219 29523 tablet_service.cc:1514] Processing DeleteTablet for tablet 750a8e8452f641d9b3db92e98d7182e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:41 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:49358
I20260926 12:42:41.112627 29763 raft_consensus.cc:2229] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:41.112769 29765 tablet_replica.cc:330] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: stopping tablet replica
I20260926 12:42:41.113016 29763 raft_consensus.cc:2258] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:41.113195 29765 raft_consensus.cc:2229] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:41.113336 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 29326
I20260926 12:42:41.114140 29765 raft_consensus.cc:2258] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:41.115352 29763 ts_tablet_manager.cc:1905] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:41.116223 29765 ts_tablet_manager.cc:1905] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:41.118093 29763 ts_tablet_manager.cc:1918] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:41.118230 29763 log.cc:1196] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/750a8e8452f641d9b3db92e98d7182e5
I20260926 12:42:41.118609 29763 ts_tablet_manager.cc:1939] T 750a8e8452f641d9b3db92e98d7182e5 P 1b1bc94a9e4d4c9b9987a67f97edac05: Deleting consensus metadata
I20260926 12:42:41.119696 29765 ts_tablet_manager.cc:1918] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:41.119802 29765 log.cc:1196] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/750a8e8452f641d9b3db92e98d7182e5
I20260926 12:42:41.119993 29250 catalog_manager.cc:4841] TS 1b1bc94a9e4d4c9b9987a67f97edac05 (127.10.224.131:34661): tablet 750a8e8452f641d9b3db92e98d7182e5 (table test-table [id=14efc5ce07b84dce92ea4bbe91241522]) successfully deleted
I20260926 12:42:41.120139 29765 ts_tablet_manager.cc:1939] T 750a8e8452f641d9b3db92e98d7182e5 P 0e68ff41c57d4d598a12d4ec300a3928: Deleting consensus metadata
I20260926 12:42:41.121546 29251 catalog_manager.cc:4841] TS 0e68ff41c57d4d598a12d4ec300a3928 (127.10.224.130:35073): tablet 750a8e8452f641d9b3db92e98d7182e5 (table test-table [id=14efc5ce07b84dce92ea4bbe91241522]) successfully deleted
W20260926 12:42:41.122076 29251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:42451 (error 108)
I20260926 12:42:41.122326 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 29460
I20260926 12:42:41.124382 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.124006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:42451 (local address 127.0.0.1:47478)
0926 12:42:41.124177 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:41.124269 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:42451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 12:42:41.124580 29251 catalog_manager.cc:4584] TS 29f1fd34068a4f269669312438aa2671 (127.10.224.129:42451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 750a8e8452f641d9b3db92e98d7182e5: Network error: Client connection negotiation failed: client connection to 127.10.224.129:42451: connect: Connection refused (error 111)
I20260926 12:42:41.128284 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 29594
I20260926 12:42:41.138409 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 29232
2026-09-26T12:42:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2530 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T12:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:41Z Disabled control of system clock
I20260926 12:42:41.169982 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:42757
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:42757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:42:41.287487 29770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:41.287853 29770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:41.287890 29770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:41.292068 29770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:41.292135 29770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:41.292150 29770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:41.292167 29770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:41.292183 29770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:41.297164 29770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:42757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:42757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:41.298273 29770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:41.299722 29770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:41.308519 29775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:41.308717 29770 server_base.cc:1034] running on GCE node
W20260926 12:42:41.308528 29778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:41.308561 29776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:41.309429 29770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:41.310679 29770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:41.311841 29770 hybrid_clock.cc:648] HybridClock initialized: now 1790426561311807 us; error 52 us; skew 500 ppm
I20260926 12:42:41.313866 29770 webserver.cc:460] Webserver started at http://127.10.224.190:42745/ using document root <none> and password file <none>
I20260926 12:42:41.314734 29770 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:41.314824 29770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:41.315073 29770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:41.317369 29770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "5987126b4c2c8462839966092934fc38"
server_key_iv: "658d9dc5a94c1b5e51687e81745799ca"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.318020 29770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "5987126b4c2c8462839966092934fc38"
server_key_iv: "658d9dc5a94c1b5e51687e81745799ca"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.322264 29770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 12:42:41.324924 29784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:41.326345 29770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:42:41.326505 29770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "5987126b4c2c8462839966092934fc38"
server_key_iv: "658d9dc5a94c1b5e51687e81745799ca"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.326635 29770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:41.344151 29770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:41.344983 29770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:41.345201 29770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:41.352689 29770 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:42757
I20260926 12:42:41.352710 29836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:42757 every 8 connection(s)
I20260926 12:42:41.353713 29770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:41.356390 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 29770
I20260926 12:42:41.356571 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:41.356864 11138 external_mini_cluster.cc:1428] Setting key 73ad38416606ae48a9b34c23031ed612
I20260926 12:42:41.357182 29837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:41.364459 29837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6: Bootstrap starting.
I20260926 12:42:41.366540 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.358251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.10.224.190:42757)
0926 12:42:41.358884 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:41.358898 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:41.358944 (+ 46us) server_negotiation.cc:407] Connection header received
0926 12:42:41.359686 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:41.359706 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:41.360152 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:41.360507 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:41.360960 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.361953 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.362888 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.363390 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.363599 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.364581 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:41.364586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:41.364619 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:41.365080 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:41.365142 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:41.365164 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:41.365301 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:41.365485 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 12:42:41.365756 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260926 12:42:41.367100 29837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:41.368067 29837 log.cc:824] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:41.370307 29837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6: No bootstrap required, opened a new log
I20260926 12:42:41.374102 29837 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42757 } }
I20260926 12:42:41.374396 29837 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:41.374439 29837 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa74f8d7a17b4b5ba0c3eee9243959e6, State: Initialized, Role: FOLLOWER
I20260926 12:42:41.374819 29837 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42757 } }
I20260926 12:42:41.374945 29837 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:41.374993 29837 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:41.375065 29837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:41.376335 29837 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42757 } }
I20260926 12:42:41.376665 29837 leader_election.cc:302] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa74f8d7a17b4b5ba0c3eee9243959e6; no voters:
I20260926 12:42:41.376967 29837 leader_election.cc:288] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:41.377133 29842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:41.377432 29842 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [term 1 LEADER]: Becoming Leader. State: Replica: aa74f8d7a17b4b5ba0c3eee9243959e6, State: Running, Role: LEADER
I20260926 12:42:41.377772 29842 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42757 } }
I20260926 12:42:41.378324 29837 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:41.379585 29844 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa74f8d7a17b4b5ba0c3eee9243959e6. Latest consensus state: current_term: 1 leader_uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42757 } } }
I20260926 12:42:41.379712 29844 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:41.380056 29843 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa74f8d7a17b4b5ba0c3eee9243959e6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 42757 } } }
I20260926 12:42:41.380164 29843 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:41.380465 29851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:41.383441 29851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:41.389662 29851 catalog_manager.cc:1347] Generated new cluster ID: aa995350d5f3421cbe6aaac3b054b50b
I20260926 12:42:41.389743 29851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:41.427076 29851 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:41.428395 29851 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:41.434595 29851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6: Generated new TSK 0
I20260926 12:42:41.435285 29851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:41.440627 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42757
--builtin_ntp_servers=127.10.224.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:42:41.555128 29861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:41.555430 29861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:41.555472 29861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:41.559736 29861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:41.559803 29861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:41.559860 29861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:42:41.564740 29861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:41.565898 29861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:41.567389 29861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:41.577687 29866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:41.577694 29867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:41.577697 29869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:41.577920 29861 server_base.cc:1034] running on GCE node
I20260926 12:42:41.578444 29861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:41.579214 29861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:41.580372 29861 hybrid_clock.cc:648] HybridClock initialized: now 1790426561580349 us; error 36 us; skew 500 ppm
I20260926 12:42:41.582436 29861 webserver.cc:460] Webserver started at http://127.10.224.129:40241/ using document root <none> and password file <none>
I20260926 12:42:41.583073 29861 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:41.583139 29861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:41.583362 29861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:41.585610 29861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "03187b3847814fa4971bdf147db11d42"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "1134f8e8de55f3f9f410d627729c86fd"
server_key_iv: "0d8542516c9e94c84cd1dc001496a4f4"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.586208 29861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "03187b3847814fa4971bdf147db11d42"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "1134f8e8de55f3f9f410d627729c86fd"
server_key_iv: "0d8542516c9e94c84cd1dc001496a4f4"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.590339 29861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:42:41.593078 29875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:41.594380 29861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:42:41.594529 29861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "03187b3847814fa4971bdf147db11d42"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "1134f8e8de55f3f9f410d627729c86fd"
server_key_iv: "0d8542516c9e94c84cd1dc001496a4f4"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.594657 29861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:41.614771 29861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:41.615581 29861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:41.615792 29861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:41.616426 29861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:41.617439 29861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:41.617509 29861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:41.617584 29861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:41.617636 29861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:41.627225 29861 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:40253
I20260926 12:42:41.627301 29988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:40253 every 8 connection(s)
I20260926 12:42:41.628340 29861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:42:41.638005 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 29861
I20260926 12:42:41.638147 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:42:41.638213 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.630437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:38297 (local address 127.10.224.190:42757)
0926 12:42:41.630595 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:41.630600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:41.631884 (+ 1284us) server_negotiation.cc:407] Connection header received
0926 12:42:41.633269 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:41.633273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:41.633332 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:41.633437 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:41.634365 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.635012 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.636247 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.636490 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.636575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.637638 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:41.637664 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:41.637670 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:41.637688 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:41.637713 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:41.637726 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:41.637820 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:41.637960 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 12:42:41.638041 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:42:41.638574 11138 external_mini_cluster.cc:1428] Setting key 3b1ed2c2f47fd9d3de3afc0d58b6acd7
I20260926 12:42:41.639245 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.630865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42757 (local address 127.10.224.129:38297)
0926 12:42:41.631689 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:41.631744 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:41.633050 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:41.633622 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:41.633630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:41.633717 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:41.634220 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:41.634232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.635134 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.635136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:41.636132 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:41.636142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.636634 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.636637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:41.636783 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.637375 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:41.637377 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:41.637417 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:41.637490 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:41.637815 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:41.637822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:41.637826 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:42:41.637922 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 12:42:41.638238 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":171,"threads_started":1}
I20260926 12:42:41.640594 29989 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42757
I20260926 12:42:41.640841 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42757
--builtin_ntp_servers=127.10.224.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:42:41.640869 29989 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:41.641541 29989 heartbeater.cc:511] Master 127.10.224.190:42757 requested a full tablet report, sending...
I20260926 12:42:41.643227 29800 ts_manager.cc:194] Registered new tserver with Master: 03187b3847814fa4971bdf147db11d42 (127.10.224.129:40253)
I20260926 12:42:41.644527 29800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.129:38297
W20260926 12:42:41.757083 29992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:41.757368 29992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:41.757411 29992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:41.761466 29992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:41.761525 29992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:41.761581 29992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:42:41.766791 29992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:41.768049 29992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:41.769465 29992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:41.779819 30000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:41.779793 29998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:41.779783 29997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:41.780143 29992 server_base.cc:1034] running on GCE node
I20260926 12:42:41.780627 29992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:41.781319 29992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:41.782502 29992 hybrid_clock.cc:648] HybridClock initialized: now 1790426561782482 us; error 32 us; skew 500 ppm
I20260926 12:42:41.784453 29992 webserver.cc:460] Webserver started at http://127.10.224.130:41601/ using document root <none> and password file <none>
I20260926 12:42:41.785061 29992 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:41.785131 29992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:41.785374 29992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:41.787590 29992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "c68611d2b60643db815bf4727d81159d"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "322296fb62bc7b21ce97f46e499cc557"
server_key_iv: "18ff3e55dd6aa2df5069a3135b6145f9"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.788192 29992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "c68611d2b60643db815bf4727d81159d"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "322296fb62bc7b21ce97f46e499cc557"
server_key_iv: "18ff3e55dd6aa2df5069a3135b6145f9"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.792143 29992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:41.794806 30006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:41.796231 29992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:41.796360 29992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "c68611d2b60643db815bf4727d81159d"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "322296fb62bc7b21ce97f46e499cc557"
server_key_iv: "18ff3e55dd6aa2df5069a3135b6145f9"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.796486 29992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:41.822994 29992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:41.823834 29992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:41.824055 29992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:41.824712 29992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:41.825722 29992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:41.825769 29992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:41.825842 29992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:41.825877 29992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:41.835618 29992 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:33807
I20260926 12:42:41.835635 30119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:33807 every 8 connection(s)
I20260926 12:42:41.836584 29992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:42:41.838434 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 29992
I20260926 12:42:41.838555 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:42:41.838876 11138 external_mini_cluster.cc:1428] Setting key 1808bcd14896510be4bdde4463b6ef7d
I20260926 12:42:41.841282 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42757
--builtin_ntp_servers=127.10.224.148:41163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:42:41.848104 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.838556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:59967 (local address 127.10.224.190:42757)
0926 12:42:41.838709 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:41.838713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:41.840069 (+ 1356us) server_negotiation.cc:407] Connection header received
0926 12:42:41.841798 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:41.841804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:41.841878 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:41.841982 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:41.843161 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.843850 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.845773 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.845998 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.846079 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.847563 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:41.847567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:41.847571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:41.847586 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:41.847611 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:41.847622 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:41.847703 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:41.847905 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 12:42:41.847967 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:42:41.849548 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:41.839037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42757 (local address 127.10.224.130:59967)
0926 12:42:41.839888 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:41.839958 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:41.841576 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:41.842151 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:41.842158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:41.842250 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:41.842970 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:41.842990 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.844002 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.844008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:41.845623 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:41.845637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:41.846169 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:41.846176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:41.846430 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:41.847165 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:41.847169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:41.847233 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:41.847339 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:41.847723 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:41.847733 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:41.847742 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:42:41.847902 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 12:42:41.848279 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":133,"threads_started":1}
I20260926 12:42:41.850909 30120 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42757
I20260926 12:42:41.851176 30120 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:41.851835 30120 heartbeater.cc:511] Master 127.10.224.190:42757 requested a full tablet report, sending...
I20260926 12:42:41.852975 29800 ts_manager.cc:194] Registered new tserver with Master: c68611d2b60643db815bf4727d81159d (127.10.224.130:33807)
I20260926 12:42:41.853603 29800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.130:59967
W20260926 12:42:41.959230 30123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:41.959623 30123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:41.959698 30123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:41.963866 30123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:41.963937 30123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:41.963997 30123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:42:41.969187 30123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:41163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:41.970371 30123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:41.971876 30123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:41.981980 30129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:41.982009 30128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:41.982191 30123 server_base.cc:1034] running on GCE node
W20260926 12:42:41.982004 30131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:41.982800 30123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:41.983503 30123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:41.984773 30123 hybrid_clock.cc:648] HybridClock initialized: now 1790426561984745 us; error 39 us; skew 500 ppm
I20260926 12:42:41.986814 30123 webserver.cc:460] Webserver started at http://127.10.224.131:34521/ using document root <none> and password file <none>
I20260926 12:42:41.987439 30123 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:41.987552 30123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:41.987795 30123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:41.989943 30123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "1da89b73056e43b69cd3f176cd71de52"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "50f4e7c6a4515cdffcc56dc8636a6263"
server_key_iv: "a60ee222379ec0404016278980b92c0a"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.990545 30123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "1da89b73056e43b69cd3f176cd71de52"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "50f4e7c6a4515cdffcc56dc8636a6263"
server_key_iv: "a60ee222379ec0404016278980b92c0a"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.994603 30123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 12:42:41.997287 30137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:41.998978 30123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:42:41.999131 30123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "1da89b73056e43b69cd3f176cd71de52"
format_stamp: "Formatted at 2026-09-26 12:42:41 on dist-test-slave-m3pj"
server_key: "50f4e7c6a4515cdffcc56dc8636a6263"
server_key_iv: "a60ee222379ec0404016278980b92c0a"
server_key_version: "encryptionkey@0"
I20260926 12:42:41.999260 30123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:42.020153 30123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:42.020938 30123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:42.021149 30123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:42.021780 30123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:42.022825 30123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:42.022909 30123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:42.023027 30123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:42.023080 30123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:42.032176 30123 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:44003
I20260926 12:42:42.032254 30250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:44003 every 8 connection(s)
I20260926 12:42:42.033146 30123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:42:42.038867 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 30123
I20260926 12:42:42.039007 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:42:42.039328 11138 external_mini_cluster.cc:1428] Setting key 7adecdec8e7b76f5d6ef47e249404849
I20260926 12:42:42.043287 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.035035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:42229 (local address 127.10.224.190:42757)
0926 12:42:42.035202 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:42.035207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:42.036414 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 12:42:42.037723 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:42.037725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:42.037776 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:42.037855 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:42.038870 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.039714 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.041169 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.041412 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.041538 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.042731 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:42.042735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:42.042737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:42.042751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:42.042772 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:42.042796 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:42.042867 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:42.043098 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 12:42:42.043156 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 12:42:42.044273 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.035497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:42757 (local address 127.10.224.131:42229)
0926 12:42:42.036214 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:42.036254 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:42.037525 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:42.038025 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:42.038032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:42.038122 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:42.038668 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.038681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.039869 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.039873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.041038 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.041049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.041584 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.041587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.041804 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.042430 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:42.042431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:42.042475 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:42.042557 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:42.042938 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:42.042947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:42.042952 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:42:42.043080 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 12:42:42.043332 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260926 12:42:42.045560 30251 heartbeater.cc:347] Connected to a master server at 127.10.224.190:42757
I20260926 12:42:42.045840 30251 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:42.046429 30251 heartbeater.cc:511] Master 127.10.224.190:42757 requested a full tablet report, sending...
I20260926 12:42:42.047612 29800 ts_manager.cc:194] Registered new tserver with Master: 1da89b73056e43b69cd3f176cd71de52 (127.10.224.131:44003)
I20260926 12:42:42.048173 29800 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.131:42229
I20260926 12:42:42.053500 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:42:42.059489 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.055379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.10.224.190:42757)
0926 12:42:42.055595 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:42.055599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:42.055722 (+ 123us) server_negotiation.cc:407] Connection header received
0926 12:42:42.055883 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:42.055886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:42.055938 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:42.056023 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:42.056294 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.056831 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.057678 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.057913 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.058026 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.058898 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:42.058901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:42.058904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:42.058918 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:42.058960 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:42.058969 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:42.059105 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:42.059254 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 12:42:42.059292 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 12:42:42.062752 29800 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:42:42.065915 29800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:42:42.081331 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.075348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:44003 (local address 127.0.0.1:45790)
0926 12:42:42.075950 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:42.075964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:42.076061 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:42.076727 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:42.076730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:42.076746 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:42.076924 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.076933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.078009 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.078013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.079629 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.079639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.080763 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.080767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.080871 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.080909 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 12:42:42.081033 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 12:42:42.081162 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":228,"spinlock_wait_cycles":1792,"thread_start_us":95,"threads_started":1}
I20260926 12:42:42.081372 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.074863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40253 (local address 127.0.0.1:53286)
0926 12:42:42.075804 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:42.075818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:42.076011 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:42.076668 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:42.076670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:42.076682 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:42.076898 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.076906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.078370 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.078373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.079757 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.079768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.081033 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.081037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.081117 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.081137 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:42:42.081207 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:42:42.081264 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":116,"thread_start_us":99,"threads_started":1}
I20260926 12:42:42.081568 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.075048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.10.224.129:40253)
0926 12:42:42.075513 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:42.075546 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:42.075901 (+ 355us) server_negotiation.cc:407] Connection header received
0926 12:42:42.076209 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:42.076214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:42.076376 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:42.076480 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:42.077032 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.078238 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.079906 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.080642 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.080787 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.080969 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:42.081266 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 12:42:42.081381 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":134,"threads_started":1}
I20260926 12:42:42.081593 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.075787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.10.224.131:44003)
0926 12:42:42.076319 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:42.076329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:42.076349 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:42:42.076418 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:42.076422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:42.076580 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:42.076911 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:42.077051 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.077772 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.079755 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.080636 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.080763 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.080946 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:42.081067 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 12:42:42.081144 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":156,"threads_started":1}
I20260926 12:42:42.082793 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.074608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33807 (local address 127.0.0.1:54522)
0926 12:42:42.075427 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:42.075479 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:42.076069 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:42.076596 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:42.076605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:42.076633 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:42.076898 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.076906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.078370 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.078373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.079776 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.079782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.082314 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.082320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.082455 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.082485 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 12:42:42.082571 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 12:42:42.082650 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":276,"spinlock_wait_cycles":11776,"thread_start_us":93,"threads_started":1}
I20260926 12:42:42.082911 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.074767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.10.224.130:33807)
0926 12:42:42.075749 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:42.075761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:42.075788 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:42:42.076208 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:42.076214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:42.076422 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:42.076580 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:42.077043 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.078251 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.079884 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.080635 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.080763 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.080946 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:42.082639 (+ 1693us) server_negotiation.cc:299] Negotiation successful
0926 12:42:42.082755 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":144,"threads_started":1}
I20260926 12:42:42.084712 29923 tablet_service.cc:1467] Processing CreateTablet for tablet 036ce3d2c3bd40d08de0078bac74da5e (DEFAULT_TABLE table=test-table [id=12686b64d6d044d99a8ad66b43f158ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:42.085220 30185 tablet_service.cc:1467] Processing CreateTablet for tablet 036ce3d2c3bd40d08de0078bac74da5e (DEFAULT_TABLE table=test-table [id=12686b64d6d044d99a8ad66b43f158ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:42.085645 30054 tablet_service.cc:1467] Processing CreateTablet for tablet 036ce3d2c3bd40d08de0078bac74da5e (DEFAULT_TABLE table=test-table [id=12686b64d6d044d99a8ad66b43f158ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:42.086259 30185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036ce3d2c3bd40d08de0078bac74da5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:42.086369 29923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036ce3d2c3bd40d08de0078bac74da5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:42.086601 30054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036ce3d2c3bd40d08de0078bac74da5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:42.092903 30268 tablet_bootstrap.cc:492] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42: Bootstrap starting.
I20260926 12:42:42.094372 30270 tablet_bootstrap.cc:492] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: Bootstrap starting.
I20260926 12:42:42.095211 30268 tablet_bootstrap.cc:654] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:42.095858 30269 tablet_bootstrap.cc:492] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: Bootstrap starting.
I20260926 12:42:42.096104 30268 log.cc:824] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:42.096372 30270 tablet_bootstrap.cc:654] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:42.097219 30270 log.cc:824] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:42.097986 30269 tablet_bootstrap.cc:654] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:42.098779 30269 log.cc:824] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:42.098999 30268 tablet_bootstrap.cc:492] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42: No bootstrap required, opened a new log
I20260926 12:42:42.099223 30268 ts_tablet_manager.cc:1397] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:42:42.099411 30270 tablet_bootstrap.cc:492] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: No bootstrap required, opened a new log
I20260926 12:42:42.099618 30270 ts_tablet_manager.cc:1397] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:42:42.100975 30269 tablet_bootstrap.cc:492] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: No bootstrap required, opened a new log
I20260926 12:42:42.101197 30269 ts_tablet_manager.cc:1397] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:42:42.103071 30270 raft_consensus.cc:348] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.103440 30270 raft_consensus.cc:374] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:42.103498 30270 raft_consensus.cc:729] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68611d2b60643db815bf4727d81159d, State: Initialized, Role: FOLLOWER
I20260926 12:42:42.103981 30270 consensus_queue.cc:260] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.104274 30268 raft_consensus.cc:348] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.104864 30268 raft_consensus.cc:374] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:42.104956 30268 raft_consensus.cc:729] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03187b3847814fa4971bdf147db11d42, State: Initialized, Role: FOLLOWER
I20260926 12:42:42.105000 30270 ts_tablet_manager.cc:1428] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:42:42.105342 30120 heartbeater.cc:503] Master 127.10.224.190:42757 was elected leader, sending a full tablet report...
I20260926 12:42:42.105412 30268 consensus_queue.cc:260] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.106191 30269 raft_consensus.cc:348] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.106432 30268 ts_tablet_manager.cc:1428] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:42:42.106530 30269 raft_consensus.cc:374] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:42.106597 30269 raft_consensus.cc:729] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da89b73056e43b69cd3f176cd71de52, State: Initialized, Role: FOLLOWER
I20260926 12:42:42.106781 29989 heartbeater.cc:503] Master 127.10.224.190:42757 was elected leader, sending a full tablet report...
I20260926 12:42:42.107038 30269 consensus_queue.cc:260] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.107658 30251 heartbeater.cc:503] Master 127.10.224.190:42757 was elected leader, sending a full tablet report...
I20260926 12:42:42.107990 30269 ts_tablet_manager.cc:1428] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260926 12:42:42.129622 29990 tablet.cc:2369] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:42.171562 30276 raft_consensus.cc:482] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:42.171842 30276 raft_consensus.cc:504] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.173195 30276 leader_election.cc:288] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c68611d2b60643db815bf4727d81159d (127.10.224.130:33807), 03187b3847814fa4971bdf147db11d42 (127.10.224.129:40253)
I20260926 12:42:42.177150 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.173604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:35077 (local address 127.10.224.130:33807)
0926 12:42:42.173731 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:42.173736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:42.173751 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:42:42.173920 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:42.173923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:42.173985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:42.174083 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:42.174389 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.174969 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.176147 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.176592 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.176721 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.176772 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:42.176957 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 12:42:42.177037 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:42:42.177150 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.173443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33807 (local address 127.10.224.131:35077)
0926 12:42:42.173664 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:42.173680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:42.173788 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:42.174113 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:42.174119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:42.174140 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:42.174261 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.174270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.175086 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.175089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.175996 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.176006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.176716 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.176719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.176821 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.176839 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:42:42.176941 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 12:42:42.177012 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 12:42:42.177265 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.173531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:52579 (local address 127.10.224.129:40253)
0926 12:42:42.173673 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:42.173680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:42.173962 (+ 282us) server_negotiation.cc:407] Connection header received
0926 12:42:42.174081 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:42.174084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:42.174135 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:42.174213 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:42.174591 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.175118 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.176286 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.176682 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.176814 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.176860 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:42.177039 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 12:42:42.177114 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:42:42.177291 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.173445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40253 (local address 127.10.224.131:52579)
0926 12:42:42.173865 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:42.173875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:42.173970 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:42.174306 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:42.174308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:42.174348 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:42.174474 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.174481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.175244 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.175247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.176155 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:42.176165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.176805 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.176808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:42.176906 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.176934 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:42:42.177022 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 12:42:42.177087 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":59,"thread_start_us":134,"threads_started":1}
I20260926 12:42:42.177892 29943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036ce3d2c3bd40d08de0078bac74da5e" candidate_uuid: "1da89b73056e43b69cd3f176cd71de52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03187b3847814fa4971bdf147db11d42" is_pre_election: true
I20260926 12:42:42.177896 30074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036ce3d2c3bd40d08de0078bac74da5e" candidate_uuid: "1da89b73056e43b69cd3f176cd71de52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68611d2b60643db815bf4727d81159d" is_pre_election: true
I20260926 12:42:42.178210 29943 raft_consensus.cc:2454] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da89b73056e43b69cd3f176cd71de52 in term 0.
I20260926 12:42:42.178210 30074 raft_consensus.cc:2454] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da89b73056e43b69cd3f176cd71de52 in term 0.
I20260926 12:42:42.178748 30139 leader_election.cc:302] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03187b3847814fa4971bdf147db11d42, 1da89b73056e43b69cd3f176cd71de52, c68611d2b60643db815bf4727d81159d; no voters:
I20260926 12:42:42.179081 30276 raft_consensus.cc:2781] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:42:42.179193 30276 raft_consensus.cc:482] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:42:42.179257 30276 raft_consensus.cc:3037] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:42.180363 30276 raft_consensus.cc:504] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.180760 30276 leader_election.cc:288] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [CANDIDATE]: Term 1 election: Requested vote from peers c68611d2b60643db815bf4727d81159d (127.10.224.130:33807), 03187b3847814fa4971bdf147db11d42 (127.10.224.129:40253)
I20260926 12:42:42.181142 29943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036ce3d2c3bd40d08de0078bac74da5e" candidate_uuid: "1da89b73056e43b69cd3f176cd71de52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03187b3847814fa4971bdf147db11d42"
I20260926 12:42:42.181285 29943 raft_consensus.cc:3037] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:42.182381 29943 raft_consensus.cc:2454] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da89b73056e43b69cd3f176cd71de52 in term 1.
I20260926 12:42:42.182713 30074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036ce3d2c3bd40d08de0078bac74da5e" candidate_uuid: "1da89b73056e43b69cd3f176cd71de52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68611d2b60643db815bf4727d81159d"
I20260926 12:42:42.182775 30138 leader_election.cc:302] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03187b3847814fa4971bdf147db11d42, 1da89b73056e43b69cd3f176cd71de52; no voters:
I20260926 12:42:42.182869 30074 raft_consensus.cc:3037] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:42.182948 30276 raft_consensus.cc:2781] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:42.183197 30276 raft_consensus.cc:686] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 1 LEADER]: Becoming Leader. State: Replica: 1da89b73056e43b69cd3f176cd71de52, State: Running, Role: LEADER
I20260926 12:42:42.183615 30276 consensus_queue.cc:237] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } }
I20260926 12:42:42.184219 30074 raft_consensus.cc:2454] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da89b73056e43b69cd3f176cd71de52 in term 1.
I20260926 12:42:42.187304 29799 catalog_manager.cc:5495] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1da89b73056e43b69cd3f176cd71de52 (127.10.224.131). New cstate: current_term: 1 leader_uuid: "1da89b73056e43b69cd3f176cd71de52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1da89b73056e43b69cd3f176cd71de52" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 44003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20260926 12:42:42.205926 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.202056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.10.224.131:44003)
0926 12:42:42.202196 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:42.202201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:42.202216 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:42:42.202348 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:42.202351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:42.202410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:42.202525 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:42:42.202792 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.203368 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.204331 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:42.204560 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:42.204673 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:42.205016 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:42:42.205439 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:42:42.205599 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:42.205696 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 12:42:42.205732 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:42:42.212083 29943 raft_consensus.cc:1261] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 1 FOLLOWER]: Refusing update from remote peer 1da89b73056e43b69cd3f176cd71de52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:42.212082 30074 raft_consensus.cc:1261] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 1 FOLLOWER]: Refusing update from remote peer 1da89b73056e43b69cd3f176cd71de52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:42.212805 30276 consensus_queue.cc:1035] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03187b3847814fa4971bdf147db11d42" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:42.213063 30278 consensus_queue.cc:1035] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68611d2b60643db815bf4727d81159d" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:42.223694 30282 mvcc.cc:205] Tried to move back new op lower bound from 7333587198812536832 to 7333587198707544064. Current Snapshot: MvccSnapshot[applied={T|T < 7333587198812536832}]
I20260926 12:42:42.224478 30283 mvcc.cc:205] Tried to move back new op lower bound from 7333587198812536832 to 7333587198707544064. Current Snapshot: MvccSnapshot[applied={T|T < 7333587198812536832}]
I20260926 12:42:42.231739 29799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:42.231935 29799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:42.234272 29799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa74f8d7a17b4b5ba0c3eee9243959e6: Sending DeleteTablet for 3 replicas of tablet 036ce3d2c3bd40d08de0078bac74da5e
I20260926 12:42:42.235183 30054 tablet_service.cc:1514] Processing DeleteTablet for tablet 036ce3d2c3bd40d08de0078bac74da5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:42 UTC) from {username='slave'} at 127.0.0.1:54522
I20260926 12:42:42.235226 29923 tablet_service.cc:1514] Processing DeleteTablet for tablet 036ce3d2c3bd40d08de0078bac74da5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:42 UTC) from {username='slave'} at 127.0.0.1:53286
I20260926 12:42:42.235841 30290 tablet_replica.cc:330] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: stopping tablet replica
I20260926 12:42:42.235878 30291 tablet_replica.cc:330] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42: stopping tablet replica
I20260926 12:42:42.236114 30185 tablet_service.cc:1514] Processing DeleteTablet for tablet 036ce3d2c3bd40d08de0078bac74da5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:42 UTC) from {username='slave'} at 127.0.0.1:45790
I20260926 12:42:42.236253 30291 raft_consensus.cc:2229] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:42.236466 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 29861
I20260926 12:42:42.236608 30291 raft_consensus.cc:2258] T 036ce3d2c3bd40d08de0078bac74da5e P 03187b3847814fa4971bdf147db11d42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:42.237638 30290 raft_consensus.cc:2229] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:42.237962 30290 raft_consensus.cc:2258] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:42.238394 30292 tablet_replica.cc:330] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: stopping tablet replica
I20260926 12:42:42.239573 30290 ts_tablet_manager.cc:1905] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:42.239930 30292 raft_consensus.cc:2229] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:42:42.240355 30292 raft_consensus.cc:2258] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:42.241887 30292 ts_tablet_manager.cc:1905] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:42.243191 30290 ts_tablet_manager.cc:1918] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:42.243299 30290 log.cc:1196] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/036ce3d2c3bd40d08de0078bac74da5e
I20260926 12:42:42.243683 30290 ts_tablet_manager.cc:1939] T 036ce3d2c3bd40d08de0078bac74da5e P c68611d2b60643db815bf4727d81159d: Deleting consensus metadata
I20260926 12:42:42.244882 30292 ts_tablet_manager.cc:1918] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:42.245055 30292 log.cc:1196] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/036ce3d2c3bd40d08de0078bac74da5e
I20260926 12:42:42.245400 30292 ts_tablet_manager.cc:1939] T 036ce3d2c3bd40d08de0078bac74da5e P 1da89b73056e43b69cd3f176cd71de52: Deleting consensus metadata
I20260926 12:42:42.245676 29786 catalog_manager.cc:4841] TS c68611d2b60643db815bf4727d81159d (127.10.224.130:33807): tablet 036ce3d2c3bd40d08de0078bac74da5e (table test-table [id=12686b64d6d044d99a8ad66b43f158ed]) successfully deleted
I20260926 12:42:42.246560 29786 catalog_manager.cc:4841] TS 1da89b73056e43b69cd3f176cd71de52 (127.10.224.131:44003): tablet 036ce3d2c3bd40d08de0078bac74da5e (table test-table [id=12686b64d6d044d99a8ad66b43f158ed]) successfully deleted
W20260926 12:42:42.247036 29785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:40253 (error 108)
I20260926 12:42:42.247287 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 29992
I20260926 12:42:42.249339 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:42.249025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:40253 (local address 127.0.0.1:53302)
0926 12:42:42.249167 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:42.249242 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:40253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 12:42:42.249575 29785 catalog_manager.cc:4584] TS 03187b3847814fa4971bdf147db11d42 (127.10.224.129:40253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 036ce3d2c3bd40d08de0078bac74da5e: Network error: Client connection negotiation failed: client connection to 127.10.224.129:40253: connect: Connection refused (error 111)
I20260926 12:42:42.254931 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 30123
I20260926 12:42:42.260901 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 29770
2026-09-26T12:42:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1121 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:42 dist-test-slave-m3pj krb5kdc[30296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:42 dist-test-slave-m3pj krb5kdc[30296](info): set up 2 sockets
Sep 26 12:42:42 dist-test-slave-m3pj krb5kdc[30296](info): commencing operation
krb5kdc: starting...
W20260926 12:42:43.312992 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:43 dist-test-slave-m3pj krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:42:43.452638 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:43283
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:43283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:43.567597 30309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:43.567879 30309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:43.567920 30309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:43.571979 30309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:43.572050 30309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:43.572072 30309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:43.572089 30309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:43.572105 30309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:43.577548 30309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:43283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:43.578637 30309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:43.579680 30309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:43.588948 30315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:43.588935 30314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:43.589012 30309 server_base.cc:1034] running on GCE node
W20260926 12:42:43.588935 30317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:43.589968 30309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:43.591123 30309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:43.592303 30309 hybrid_clock.cc:648] HybridClock initialized: now 1790426563592264 us; error 62 us; skew 500 ppm
Sep 26 12:42:43 dist-test-slave-m3pj krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426563, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:43.595608 30309 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:42:43.596660 30309 webserver.cc:460] Webserver started at http://127.10.224.190:36437/ using document root <none> and password file <none>
I20260926 12:42:43.597276 30309 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:43.597343 30309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:43.597605 30309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:43.599921 30309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "6d9e981461f346c78ec35e149e9cf90f"
format_stamp: "Formatted at 2026-09-26 12:42:43 on dist-test-slave-m3pj"
server_key: "81be929e88c353e431ae9eec92d02b68"
server_key_iv: "47b762c73e59f836a3c8f1e52ac8ff22"
server_key_version: "encryptionkey@0"
I20260926 12:42:43.600534 30309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "6d9e981461f346c78ec35e149e9cf90f"
format_stamp: "Formatted at 2026-09-26 12:42:43 on dist-test-slave-m3pj"
server_key: "81be929e88c353e431ae9eec92d02b68"
server_key_iv: "47b762c73e59f836a3c8f1e52ac8ff22"
server_key_version: "encryptionkey@0"
I20260926 12:42:43.604600 30309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:43.607191 30324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:43.608563 30309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:42:43.608698 30309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "6d9e981461f346c78ec35e149e9cf90f"
format_stamp: "Formatted at 2026-09-26 12:42:43 on dist-test-slave-m3pj"
server_key: "81be929e88c353e431ae9eec92d02b68"
server_key_iv: "47b762c73e59f836a3c8f1e52ac8ff22"
server_key_version: "encryptionkey@0"
I20260926 12:42:43.608816 30309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:43.620978 30309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:43.624433 30309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:43.624655 30309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:43.632093 30309 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:43283
I20260926 12:42:43.632117 30376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:43283 every 8 connection(s)
I20260926 12:42:43.633119 30309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:43.636127 30377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:43.639578 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 30309
I20260926 12:42:43.639735 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:43.640059 11138 external_mini_cluster.cc:1428] Setting key ab94b8b4a2e979ce1b84b4c6b8fa0142
I20260926 12:42:43.642907 30377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f: Bootstrap starting.
I20260926 12:42:43.645275 30377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:43.646075 30377 log.cc:824] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:43.648046 30377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f: No bootstrap required, opened a new log
Sep 26 12:42:43 dist-test-slave-m3pj krb5kdc[30296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:43.651916 30377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9e981461f346c78ec35e149e9cf90f" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43283 } }
I20260926 12:42:43.652156 30377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:43.652191 30377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d9e981461f346c78ec35e149e9cf90f, State: Initialized, Role: FOLLOWER
I20260926 12:42:43.652725 30377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9e981461f346c78ec35e149e9cf90f" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43283 } }
I20260926 12:42:43.652907 30377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:43.652999 30377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:43.653102 30377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:43.654090 30377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9e981461f346c78ec35e149e9cf90f" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43283 } }
I20260926 12:42:43.654479 30377 leader_election.cc:302] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d9e981461f346c78ec35e149e9cf90f; no voters:
I20260926 12:42:43.654911 30377 leader_election.cc:288] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:43.654979 30382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:43.655196 30382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [term 1 LEADER]: Becoming Leader. State: Replica: 6d9e981461f346c78ec35e149e9cf90f, State: Running, Role: LEADER
I20260926 12:42:43.655571 30382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9e981461f346c78ec35e149e9cf90f" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43283 } }
I20260926 12:42:43.655859 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:43.641771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.10.224.190:43283)
0926 12:42:43.642295 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:43.642311 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:43.642349 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:42:43.643092 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:43.643113 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:43.643464 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:43.643885 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:43.644712 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:43.645782 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:43.646743 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:43.647232 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:43.647409 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:43.650293 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:43.650298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:43.650313 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:43.650342 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:43.652354 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:43.652852 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:43.652889 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:43.652897 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:43.653097 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:43.653403 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:43.653406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:43.653408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:43.653844 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:43.654064 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:43.654371 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 12:42:43.654738 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":165,"threads_started":1}
I20260926 12:42:43.656351 30377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:43.657335 30384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d9e981461f346c78ec35e149e9cf90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9e981461f346c78ec35e149e9cf90f" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43283 } } }
I20260926 12:42:43.657470 30384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:43.657840 30383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9e981461f346c78ec35e149e9cf90f. Latest consensus state: current_term: 1 leader_uuid: "6d9e981461f346c78ec35e149e9cf90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d9e981461f346c78ec35e149e9cf90f" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 43283 } } }
I20260926 12:42:43.657929 30383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:43.658627 30391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:43.661844 30391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:43.667582 30391 catalog_manager.cc:1347] Generated new cluster ID: a88d9c3c61cb452c983c97bd71f857cc
I20260926 12:42:43.667668 30391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:43.675320 30391 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:43.676416 30391 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:43.686007 30391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f: Generated new TSK 0
I20260926 12:42:43.686826 30391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.224.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
Entry for principal kudu/127.10.224.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab.
I20260926 12:42:43.720232 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43283
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:43.838376 30403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:43.838676 30403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:43.838722 30403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:43.843014 30403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:43.843109 30403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:43.843174 30403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:42:43.849485 30403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.129.keytab
--principal=kudu/127.10.224.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:43.850653 30403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:43.851702 30403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:43.861783 30409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:43.861912 30408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:43.861783 30411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:43.862170 30403 server_base.cc:1034] running on GCE node
I20260926 12:42:43.862605 30403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:43.863281 30403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:43.864437 30403 hybrid_clock.cc:648] HybridClock initialized: now 1790426563864393 us; error 53 us; skew 500 ppm
Sep 26 12:42:43 dist-test-slave-m3pj krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426563, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:43.867257 30403 init.cc:377] Logged in from keytab as kudu/127.10.224.129@KRBTEST.COM (short username kudu)
I20260926 12:42:43.868216 30403 webserver.cc:460] Webserver started at http://127.10.224.129:42163/ using document root <none> and password file <none>
I20260926 12:42:43.868793 30403 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:43.868855 30403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:43.869107 30403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:43.871280 30403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "acd2626a6b9e4feea8b432cb564977f6"
format_stamp: "Formatted at 2026-09-26 12:42:43 on dist-test-slave-m3pj"
server_key: "651294181bc7fdaf23ef56d20becdb1d"
server_key_iv: "6ff174b157bf3d74fbe46ae879d13749"
server_key_version: "encryptionkey@0"
I20260926 12:42:43.871969 30403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "acd2626a6b9e4feea8b432cb564977f6"
format_stamp: "Formatted at 2026-09-26 12:42:43 on dist-test-slave-m3pj"
server_key: "651294181bc7fdaf23ef56d20becdb1d"
server_key_iv: "6ff174b157bf3d74fbe46ae879d13749"
server_key_version: "encryptionkey@0"
I20260926 12:42:43.875829 30403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:43.878531 30418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:43.879944 30403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:42:43.880069 30403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "acd2626a6b9e4feea8b432cb564977f6"
format_stamp: "Formatted at 2026-09-26 12:42:43 on dist-test-slave-m3pj"
server_key: "651294181bc7fdaf23ef56d20becdb1d"
server_key_iv: "6ff174b157bf3d74fbe46ae879d13749"
server_key_version: "encryptionkey@0"
I20260926 12:42:43.880178 30403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:43.895064 30403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:43.898216 30403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:43.898425 30403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:43.899040 30403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:43.899976 30403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:43.900024 30403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:43.900096 30403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:43.900127 30403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:43.909392 30403 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:43877
I20260926 12:42:43.909409 30531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:43877 every 8 connection(s)
I20260926 12:42:43.910349 30403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:42:43.917743 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 30403
I20260926 12:42:43.917891 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:42:43.918203 11138 external_mini_cluster.cc:1428] Setting key 4f38be3231edd78509c57cf821c6f137
Sep 26 12:42:43 dist-test-slave-m3pj krb5kdc[30296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426563, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.129@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:43.926283 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:43.912247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:42845 (local address 127.10.224.190:43283)
0926 12:42:43.912411 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:43.912415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:43.913433 (+ 1018us) server_negotiation.cc:407] Connection header received
0926 12:42:43.914772 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:43.914776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:43.914833 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:43.914941 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:43.916241 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:43.916786 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:43.918243 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:43.918548 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:43.918683 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:43.921834 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:43.921841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:43.921846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:43.921881 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:43.923704 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:43.924326 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:43.924333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:43.924338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:43.924455 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:43.924895 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:43.924901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:43.924905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:43.925181 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:43.925299 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:43.925849 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 12:42:43.926024 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:42:43.927045 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:43.912747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43283 (local address 127.10.224.129:42845)
0926 12:42:43.913244 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:43.913287 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:43.914551 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:43.915123 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:43.915130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:43.915496 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:43.916099 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:43.916112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:43.916936 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:43.916939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:43.918111 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:43.918123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:43.918726 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:43.918729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:43.918948 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:43.919719 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:43.919736 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:43.921626 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:43.923864 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:43.923870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:43.923887 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:43.924207 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:43.924587 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:43.924592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:43.924593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:43.924788 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:43.925293 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:43.925299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:43.925587 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 12:42:43.925758 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 12:42:43.926066 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260926 12:42:43.928442 30532 heartbeater.cc:347] Connected to a master server at 127.10.224.190:43283
I20260926 12:42:43.928738 30532 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.224.130@KRBTEST.COM; defaulting to no policy
I20260926 12:42:43.929423 30532 heartbeater.cc:511] Master 127.10.224.190:43283 requested a full tablet report, sending...
I20260926 12:42:43.931139 30341 ts_manager.cc:194] Registered new tserver with Master: acd2626a6b9e4feea8b432cb564977f6 (127.10.224.129:43877)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.130@KRBTEST.COM" created.
I20260926 12:42:43.932746 30341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.129@KRBTEST.COM'} at 127.10.224.129:42845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
Entry for principal kudu/127.10.224.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab.
I20260926 12:42:43.951081 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43283
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:44.074278 30537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:44.074594 30537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:44.074640 30537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:44.079021 30537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:44.079111 30537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:44.079176 30537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:42:44.085271 30537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.130.keytab
--principal=kudu/127.10.224.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:44.086513 30537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:44.087702 30537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:44.098270 30545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:44.098310 30543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:44.098270 30542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:44.098493 30537 server_base.cc:1034] running on GCE node
I20260926 12:42:44.099108 30537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:44.099928 30537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:44.101130 30537 hybrid_clock.cc:648] HybridClock initialized: now 1790426564101109 us; error 32 us; skew 500 ppm
Sep 26 12:42:44 dist-test-slave-m3pj krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426564, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:44.104228 30537 init.cc:377] Logged in from keytab as kudu/127.10.224.130@KRBTEST.COM (short username kudu)
I20260926 12:42:44.105254 30537 webserver.cc:460] Webserver started at http://127.10.224.130:32927/ using document root <none> and password file <none>
I20260926 12:42:44.105877 30537 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:44.105981 30537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:44.106230 30537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:44.108644 30537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "94c2dd2be31744a1bc137b3ee85aaa44"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "cd3770651c3f569e4b5b3d5166b5defd"
server_key_iv: "7481281cf3bdd4d012d277eceb2efbb8"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.109274 30537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "94c2dd2be31744a1bc137b3ee85aaa44"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "cd3770651c3f569e4b5b3d5166b5defd"
server_key_iv: "7481281cf3bdd4d012d277eceb2efbb8"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.113732 30537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 12:42:44.116608 30552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:44.118072 30537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:42:44.118290 30537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "94c2dd2be31744a1bc137b3ee85aaa44"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "cd3770651c3f569e4b5b3d5166b5defd"
server_key_iv: "7481281cf3bdd4d012d277eceb2efbb8"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.118451 30537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:44.151050 30537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:44.154947 30537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:44.155194 30537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:44.155922 30537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:44.156852 30537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:44.156914 30537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:44.156958 30537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:44.157020 30537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:44.167310 30537 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:33891
I20260926 12:42:44.167332 30665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:33891 every 8 connection(s)
I20260926 12:42:44.168412 30537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:42:44.168820 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 30537
I20260926 12:42:44.168972 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:42:44.169349 11138 external_mini_cluster.cc:1428] Setting key e71d5a4f36157cb46171177b4c9ff4d7
Sep 26 12:42:44 dist-test-slave-m3pj krb5kdc[30296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426564, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.130@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.131@KRBTEST.COM" created.
I20260926 12:42:44.184406 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.170555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:33579 (local address 127.10.224.190:43283)
0926 12:42:44.170726 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.170730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.171987 (+ 1257us) server_negotiation.cc:407] Connection header received
0926 12:42:44.173372 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.173376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.173434 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.173573 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:44.174769 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.175395 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.176591 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.176826 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.176951 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.179965 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:44.179968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:44.179970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:44.179998 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:44.181575 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:44.182158 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:44.182161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:44.182162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:44.182241 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:44.182697 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:44.182704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:44.182709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:44.182972 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:44.183063 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.183997 (+ 934us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.184180 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:42:44.186059 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.171233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43283 (local address 127.10.224.130:33579)
0926 12:42:44.171809 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:44.171887 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:44.173153 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:44.173763 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:44.173770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:44.174135 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:44.174624 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.174637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.175553 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.175562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.176458 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.176468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.176987 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.176991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.177168 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.177874 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:44.177891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:44.179772 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:44.181736 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:44.181742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:44.181790 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:44.182047 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:44.182345 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:44.182348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:44.182350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:44.182578 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:44.183117 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:44.183129 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:44.183601 (+ 472us) client_negotiation.cc:769] Sending connection context
0926 12:42:44.184471 (+ 870us) client_negotiation.cc:240] Negotiation successful
0926 12:42:44.184832 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":204,"threads_started":1}
I20260926 12:42:44.187455 30666 heartbeater.cc:347] Connected to a master server at 127.10.224.190:43283
I20260926 12:42:44.187774 30666 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:44.188339 30666 heartbeater.cc:511] Master 127.10.224.190:43283 requested a full tablet report, sending...
I20260926 12:42:44.189528 30341 ts_manager.cc:194] Registered new tserver with Master: 94c2dd2be31744a1bc137b3ee85aaa44 (127.10.224.130:33891)
I20260926 12:42:44.190141 30341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.130@KRBTEST.COM'} at 127.10.224.130:33579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
Entry for principal kudu/127.10.224.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab.
I20260926 12:42:44.203395 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43283
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:44.323153 30671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:44.323490 30671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:44.323642 30671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:44.327903 30671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:44.328006 30671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:44.328109 30671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:42:44.333693 30671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.131.keytab
--principal=kudu/127.10.224.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:44.334851 30671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:44.335938 30671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:44.346005 30677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:44.346120 30679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:44.346015 30676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:44.346220 30671 server_base.cc:1034] running on GCE node
I20260926 12:42:44.346736 30671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:44.347424 30671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:44.348598 30671 hybrid_clock.cc:648] HybridClock initialized: now 1790426564348578 us; error 31 us; skew 500 ppm
Sep 26 12:42:44 dist-test-slave-m3pj krb5kdc[30296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426564, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:44.351567 30671 init.cc:377] Logged in from keytab as kudu/127.10.224.131@KRBTEST.COM (short username kudu)
I20260926 12:42:44.352558 30671 webserver.cc:460] Webserver started at http://127.10.224.131:46705/ using document root <none> and password file <none>
I20260926 12:42:44.353174 30671 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:44.353266 30671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:44.353485 30671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:44.355703 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "883c643b591c4893b38b5c32e43a8f42"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "1c7fad7ceebcbeda00ce8ec75215fbca"
server_key_iv: "1606ae72391228e6a20e367737bf7851"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.356370 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "883c643b591c4893b38b5c32e43a8f42"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "1c7fad7ceebcbeda00ce8ec75215fbca"
server_key_iv: "1606ae72391228e6a20e367737bf7851"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.360679 30671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:44.363346 30686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:44.364827 30671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:44.364980 30671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "883c643b591c4893b38b5c32e43a8f42"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "1c7fad7ceebcbeda00ce8ec75215fbca"
server_key_iv: "1606ae72391228e6a20e367737bf7851"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.365113 30671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:44.377202 30671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:44.380363 30671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:44.380581 30671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:44.381182 30671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:44.382022 30671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:44.382091 30671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:44.382172 30671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:44.382221 30671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:44.391566 30671 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:36943
I20260926 12:42:44.391572 30799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:36943 every 8 connection(s)
I20260926 12:42:44.392513 30671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:42:44.400930 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 30671
I20260926 12:42:44.401101 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:42:44.401472 11138 external_mini_cluster.cc:1428] Setting key 36558756c49694f02ae4a4ed783fd1e0
Sep 26 12:42:44 dist-test-slave-m3pj krb5kdc[30296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426564, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.131@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:42:44.407181 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.394365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:44195 (local address 127.10.224.190:43283)
0926 12:42:44.394520 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.394523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.395315 (+ 792us) server_negotiation.cc:407] Connection header received
0926 12:42:44.396710 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.396714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.396773 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.396849 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:44.398158 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.398731 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.399929 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.400158 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.400241 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.403322 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:44.403326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:44.403329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:44.403360 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:44.404965 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:44.405505 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:44.405508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:44.405509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:44.405590 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:44.405991 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:44.405994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:44.405995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:44.406219 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:44.406298 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.406943 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.407053 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:42:44.408221 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.394697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:43283 (local address 127.10.224.131:44195)
0926 12:42:44.395147 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:44.395191 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:44.396493 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:44.397025 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:44.397034 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:44.397463 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:44.398018 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.398033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.398847 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.398850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.399822 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.399831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.400292 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.400294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.400475 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.401232 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:42:44.401256 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:44.403163 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:44.405074 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:44.405080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:44.405094 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:44.405371 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:44.405710 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:42:44.405713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:42:44.405715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:42:44.405887 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:42:44.406397 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:44.406403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:44.406686 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 12:42:44.406877 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 12:42:44.407179 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260926 12:42:44.409476 30800 heartbeater.cc:347] Connected to a master server at 127.10.224.190:43283
I20260926 12:42:44.409735 30800 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:44.410248 30800 heartbeater.cc:511] Master 127.10.224.190:43283 requested a full tablet report, sending...
I20260926 12:42:44.411336 30341 ts_manager.cc:194] Registered new tserver with Master: 883c643b591c4893b38b5c32e43a8f42 (127.10.224.131:36943)
I20260926 12:42:44.412050 30341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.224.131@KRBTEST.COM'} at 127.10.224.131:44195
I20260926 12:42:44.415757 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:42:44.425324 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.417799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.10.224.190:43283)
0926 12:42:44.417933 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.417938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.418091 (+ 153us) server_negotiation.cc:407] Connection header received
0926 12:42:44.418194 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.418196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.418237 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.418298 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:44.418974 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.419633 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.420520 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.420728 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.420807 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.422186 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:44.422189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:44.422191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:44.422213 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:44.423760 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:44.424180 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:44.424183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:44.424184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:44.424257 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:44.424500 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:44.424502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:44.424503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:44.424712 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:44.424813 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.425016 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.425116 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:42:44.428483 30341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:42:44.432152 30341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:42:44.448066 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.442003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.10.224.130:33891)
0926 12:42:44.442377 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.442391 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.442726 (+ 335us) server_negotiation.cc:407] Connection header received
0926 12:42:44.442981 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.442987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.443232 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.443392 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:44.443847 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.444723 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.446635 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.447231 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.447385 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.447577 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.447789 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.447899 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
I20260926 12:42:44.448060 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.441627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33891 (local address 127.0.0.1:34148)
0926 12:42:44.442578 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:44.442638 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:44.442782 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:44.443618 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:44.443621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:44.443637 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:44.443732 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.443738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.444859 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.444864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.446493 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.446504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.447383 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.447387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.447575 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.447607 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 12:42:44.447750 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 12:42:44.447905 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":169,"thread_start_us":117,"threads_started":1}
I20260926 12:42:44.448074 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.441517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:36943 (local address 127.0.0.1:41928)
0926 12:42:44.442578 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:44.442638 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:44.442782 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:44.443399 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:44.443405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:44.443428 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:44.443620 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.443627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.445111 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.445115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.446493 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.446504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.447384 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.447387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.447559 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.447605 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 12:42:44.447727 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 12:42:44.447905 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":134,"thread_start_us":97,"threads_started":1}
I20260926 12:42:44.448810 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.441642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.10.224.131:36943)
0926 12:42:44.442367 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.442381 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.442885 (+ 504us) server_negotiation.cc:407] Connection header received
0926 12:42:44.442985 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.442990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.443222 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.443394 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:44.443740 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.444975 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.446648 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.447235 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.447398 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.447576 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.448553 (+ 977us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.448668 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":111,"threads_started":1}
I20260926 12:42:44.449383 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.442079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:43877 (local address 127.0.0.1:44638)
0926 12:42:44.443274 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:44.443292 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:44.443383 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:44.443935 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:44.443938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:44.443956 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:44.444037 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.444043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.445304 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.445311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.446503 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.446513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.448983 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.448988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.449092 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.449123 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 12:42:44.449190 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:42:44.449256 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":135,"thread_start_us":96,"threads_started":1}
I20260926 12:42:44.449944 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.442513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.10.224.129:43877)
0926 12:42:44.443079 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.443085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.443489 (+ 404us) server_negotiation.cc:407] Connection header received
0926 12:42:44.443600 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.443604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.443800 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.443940 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:44.444157 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.445195 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.447637 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.448760 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.449538 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.449670 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.449719 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.449808 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":93,"threads_started":1}
I20260926 12:42:44.451423 30600 tablet_service.cc:1467] Processing CreateTablet for tablet caf70bb252ca44e5ac85042e95110957 (DEFAULT_TABLE table=test-table [id=f32caa3ad2a94e0ea51fa7e57493b50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:44.451592 30734 tablet_service.cc:1467] Processing CreateTablet for tablet caf70bb252ca44e5ac85042e95110957 (DEFAULT_TABLE table=test-table [id=f32caa3ad2a94e0ea51fa7e57493b50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:44.452440 30600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf70bb252ca44e5ac85042e95110957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:44.452618 30734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf70bb252ca44e5ac85042e95110957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:44.452775 30466 tablet_service.cc:1467] Processing CreateTablet for tablet caf70bb252ca44e5ac85042e95110957 (DEFAULT_TABLE table=test-table [id=f32caa3ad2a94e0ea51fa7e57493b50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:44.453715 30466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf70bb252ca44e5ac85042e95110957. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:44.459357 30817 tablet_bootstrap.cc:492] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Bootstrap starting.
I20260926 12:42:44.461166 30819 tablet_bootstrap.cc:492] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6: Bootstrap starting.
I20260926 12:42:44.461614 30818 tablet_bootstrap.cc:492] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Bootstrap starting.
I20260926 12:42:44.461988 30817 tablet_bootstrap.cc:654] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:44.463370 30817 log.cc:824] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:44.463546 30818 tablet_bootstrap.cc:654] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:44.463598 30819 tablet_bootstrap.cc:654] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:44.464424 30819 log.cc:824] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:44.464774 30818 log.cc:824] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:44.465767 30817 tablet_bootstrap.cc:492] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: No bootstrap required, opened a new log
I20260926 12:42:44.465986 30817 ts_tablet_manager.cc:1397] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260926 12:42:44.466459 30819 tablet_bootstrap.cc:492] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6: No bootstrap required, opened a new log
I20260926 12:42:44.466665 30819 ts_tablet_manager.cc:1397] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 12:42:44.466902 30818 tablet_bootstrap.cc:492] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: No bootstrap required, opened a new log
I20260926 12:42:44.467069 30818 ts_tablet_manager.cc:1397] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:42:44.469419 30817 raft_consensus.cc:348] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
I20260926 12:42:44.469734 30817 raft_consensus.cc:374] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:44.469792 30817 raft_consensus.cc:729] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c2dd2be31744a1bc137b3ee85aaa44, State: Initialized, Role: FOLLOWER
I20260926 12:42:44.470290 30817 consensus_queue.cc:260] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
I20260926 12:42:44.471144 30818 raft_consensus.cc:348] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
I20260926 12:42:44.471400 30817 ts_tablet_manager.cc:1428] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 12:42:44.471415 30818 raft_consensus.cc:374] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:44.471191 30819 raft_consensus.cc:348] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
I20260926 12:42:44.471470 30818 raft_consensus.cc:729] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 883c643b591c4893b38b5c32e43a8f42, State: Initialized, Role: FOLLOWER
I20260926 12:42:44.471491 30819 raft_consensus.cc:374] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:44.471563 30819 raft_consensus.cc:729] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd2626a6b9e4feea8b432cb564977f6, State: Initialized, Role: FOLLOWER
I20260926 12:42:44.471789 30666 heartbeater.cc:503] Master 127.10.224.190:43283 was elected leader, sending a full tablet report...
I20260926 12:42:44.471974 30818 consensus_queue.cc:260] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
I20260926 12:42:44.471974 30819 consensus_queue.cc:260] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
I20260926 12:42:44.472882 30800 heartbeater.cc:503] Master 127.10.224.190:43283 was elected leader, sending a full tablet report...
I20260926 12:42:44.472961 30532 heartbeater.cc:503] Master 127.10.224.190:43283 was elected leader, sending a full tablet report...
I20260926 12:42:44.473052 30818 ts_tablet_manager.cc:1428] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:42:44.473969 30819 ts_tablet_manager.cc:1428] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 12:42:44.627857 30824 raft_consensus.cc:482] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:44.628122 30824 raft_consensus.cc:504] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
I20260926 12:42:44.629372 30824 leader_election.cc:288] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 883c643b591c4893b38b5c32e43a8f42 (127.10.224.131:36943), 94c2dd2be31744a1bc137b3ee85aaa44 (127.10.224.130:33891)
I20260926 12:42:44.634886 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.629757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:38383 (local address 127.10.224.130:33891)
0926 12:42:44.629903 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.629908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.630336 (+ 428us) server_negotiation.cc:407] Connection header received
0926 12:42:44.630404 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.630410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.630499 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.630589 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:44.631005 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.631807 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.633838 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.634239 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.634373 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.634440 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.634655 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.634752 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:42:44.635002 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.629871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:59839 (local address 127.10.224.131:36943)
0926 12:42:44.629996 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.630001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.630219 (+ 218us) server_negotiation.cc:407] Connection header received
0926 12:42:44.630509 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.630513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.630577 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.630648 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:44.631003 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.631852 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.633814 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.634454 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.634584 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.634629 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.634803 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.634884 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:42:44.635350 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.629688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:36943 (local address 127.10.224.129:59839)
0926 12:42:44.630120 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:44.630138 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:44.630309 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:44.630704 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:44.630707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:44.630718 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:44.630892 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.630897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.631962 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.631965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.633671 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.633683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.634576 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.634582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.634684 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.634704 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:42:44.634773 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 12:42:44.634847 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":204,"thread_start_us":147,"threads_started":1}
I20260926 12:42:44.635349 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.629686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:33891 (local address 127.10.224.129:38383)
0926 12:42:44.630228 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:44.630243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:44.630324 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:44.630615 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:44.630618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:44.630636 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:44.630869 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.630879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.631960 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.631963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.633674 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:44.633683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.634373 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.634377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:44.634480 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.634531 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 12:42:44.635126 (+ 595us) client_negotiation.cc:240] Negotiation successful
0926 12:42:44.635208 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":478,"thread_start_us":144,"threads_started":1}
I20260926 12:42:44.636107 30754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf70bb252ca44e5ac85042e95110957" candidate_uuid: "acd2626a6b9e4feea8b432cb564977f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "883c643b591c4893b38b5c32e43a8f42" is_pre_election: true
I20260926 12:42:44.636094 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf70bb252ca44e5ac85042e95110957" candidate_uuid: "acd2626a6b9e4feea8b432cb564977f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" is_pre_election: true
I20260926 12:42:44.636427 30620 raft_consensus.cc:2454] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd2626a6b9e4feea8b432cb564977f6 in term 0.
I20260926 12:42:44.636427 30754 raft_consensus.cc:2454] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd2626a6b9e4feea8b432cb564977f6 in term 0.
I20260926 12:42:44.636974 30419 leader_election.cc:302] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 883c643b591c4893b38b5c32e43a8f42, 94c2dd2be31744a1bc137b3ee85aaa44, acd2626a6b9e4feea8b432cb564977f6; no voters:
I20260926 12:42:44.637422 30824 raft_consensus.cc:2781] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:42:44.637503 30824 raft_consensus.cc:482] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:42:44.637534 30824 raft_consensus.cc:3037] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:44.638612 30824 raft_consensus.cc:504] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
I20260926 12:42:44.639107 30824 leader_election.cc:288] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [CANDIDATE]: Term 1 election: Requested vote from peers 883c643b591c4893b38b5c32e43a8f42 (127.10.224.131:36943), 94c2dd2be31744a1bc137b3ee85aaa44 (127.10.224.130:33891)
I20260926 12:42:44.639411 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf70bb252ca44e5ac85042e95110957" candidate_uuid: "acd2626a6b9e4feea8b432cb564977f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94c2dd2be31744a1bc137b3ee85aaa44"
I20260926 12:42:44.639482 30754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf70bb252ca44e5ac85042e95110957" candidate_uuid: "acd2626a6b9e4feea8b432cb564977f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "883c643b591c4893b38b5c32e43a8f42"
I20260926 12:42:44.639583 30620 raft_consensus.cc:3037] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:44.639621 30754 raft_consensus.cc:3037] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:44.640925 30620 raft_consensus.cc:2454] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd2626a6b9e4feea8b432cb564977f6 in term 1.
I20260926 12:42:44.640925 30754 raft_consensus.cc:2454] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd2626a6b9e4feea8b432cb564977f6 in term 1.
I20260926 12:42:44.641247 30419 leader_election.cc:302] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94c2dd2be31744a1bc137b3ee85aaa44, acd2626a6b9e4feea8b432cb564977f6; no voters:
I20260926 12:42:44.641489 30824 raft_consensus.cc:2781] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:44.641743 30824 raft_consensus.cc:686] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [term 1 LEADER]: Becoming Leader. State: Replica: acd2626a6b9e4feea8b432cb564977f6, State: Running, Role: LEADER
I20260926 12:42:44.642148 30824 consensus_queue.cc:237] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } }
W20260926 12:42:44.643458 30801 tablet.cc:2369] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:44.645766 30340 catalog_manager.cc:5495] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd2626a6b9e4feea8b432cb564977f6 (127.10.224.129). New cstate: current_term: 1 leader_uuid: "acd2626a6b9e4feea8b432cb564977f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acd2626a6b9e4feea8b432cb564977f6" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 43877 } health_report { overall_health: HEALTHY } } }
W20260926 12:42:44.661687 30533 tablet.cc:2369] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:42:44.670053 30667 tablet.cc:2369] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:44.684760 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.681016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.10.224.129:43877)
0926 12:42:44.681144 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.681148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.681162 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:42:44.681260 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.681262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.681320 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.681412 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:42:44.681676 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.682207 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.683076 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.683296 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.683388 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.683780 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:42:44.684251 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:42:44.684349 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.684478 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.684540 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:42:44.690877 30754 raft_consensus.cc:1261] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 1 FOLLOWER]: Refusing update from remote peer acd2626a6b9e4feea8b432cb564977f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:44.690876 30620 raft_consensus.cc:1261] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 1 FOLLOWER]: Refusing update from remote peer acd2626a6b9e4feea8b432cb564977f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:44.691629 30824 consensus_queue.cc:1035] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "883c643b591c4893b38b5c32e43a8f42" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 36943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:44.691833 30828 consensus_queue.cc:1035] T caf70bb252ca44e5ac85042e95110957 P acd2626a6b9e4feea8b432cb564977f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94c2dd2be31744a1bc137b3ee85aaa44" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 33891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:44.709131 30340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:44.709293 30340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:44.711632 30340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d9e981461f346c78ec35e149e9cf90f: Sending DeleteTablet for 3 replicas of tablet caf70bb252ca44e5ac85042e95110957
I20260926 12:42:44.712607 30600 tablet_service.cc:1514] Processing DeleteTablet for tablet caf70bb252ca44e5ac85042e95110957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:44 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:34148
I20260926 12:42:44.712638 30734 tablet_service.cc:1514] Processing DeleteTablet for tablet caf70bb252ca44e5ac85042e95110957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:44 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:41928
I20260926 12:42:44.713187 30466 tablet_service.cc:1514] Processing DeleteTablet for tablet caf70bb252ca44e5ac85042e95110957 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:44 UTC) from {username='kudu', principal='kudu/127.10.224.190@KRBTEST.COM'} at 127.0.0.1:44638
I20260926 12:42:44.713429 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 30403
I20260926 12:42:44.714934 30841 tablet_replica.cc:330] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: stopping tablet replica
I20260926 12:42:44.715099 30840 tablet_replica.cc:330] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: stopping tablet replica
I20260926 12:42:44.715324 30841 raft_consensus.cc:2229] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:44.715374 30840 raft_consensus.cc:2229] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:44.715716 30840 raft_consensus.cc:2258] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:44.715718 30841 raft_consensus.cc:2258] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:44.717314 30840 ts_tablet_manager.cc:1905] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:44.717325 30841 ts_tablet_manager.cc:1905] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:44.719908 30840 ts_tablet_manager.cc:1918] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:44.720042 30840 log.cc:1196] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/caf70bb252ca44e5ac85042e95110957
I20260926 12:42:44.720335 30840 ts_tablet_manager.cc:1939] T caf70bb252ca44e5ac85042e95110957 P 94c2dd2be31744a1bc137b3ee85aaa44: Deleting consensus metadata
I20260926 12:42:44.720804 30841 ts_tablet_manager.cc:1918] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:44.720916 30841 log.cc:1196] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/caf70bb252ca44e5ac85042e95110957
I20260926 12:42:44.721238 30841 ts_tablet_manager.cc:1939] T caf70bb252ca44e5ac85042e95110957 P 883c643b591c4893b38b5c32e43a8f42: Deleting consensus metadata
I20260926 12:42:44.722031 30325 catalog_manager.cc:4841] TS 94c2dd2be31744a1bc137b3ee85aaa44 (127.10.224.130:33891): tablet caf70bb252ca44e5ac85042e95110957 (table test-table [id=f32caa3ad2a94e0ea51fa7e57493b50c]) successfully deleted
W20260926 12:42:44.722177 30326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:43877 (error 108)
I20260926 12:42:44.722453 30327 catalog_manager.cc:4841] TS 883c643b591c4893b38b5c32e43a8f42 (127.10.224.131:36943): tablet caf70bb252ca44e5ac85042e95110957 (table test-table [id=f32caa3ad2a94e0ea51fa7e57493b50c]) successfully deleted
I20260926 12:42:44.722569 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 30537
I20260926 12:42:44.724583 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.724252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:43877 (local address 127.0.0.1:44658)
0926 12:42:44.724407 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:44.724504 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 12:42:44.724762 30326 catalog_manager.cc:4584] TS acd2626a6b9e4feea8b432cb564977f6 (127.10.224.129:43877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet caf70bb252ca44e5ac85042e95110957: Network error: Client connection negotiation failed: client connection to 127.10.224.129:43877: connect: Connection refused (error 111)
I20260926 12:42:44.729230 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 30671
I20260926 12:42:44.734958 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 30309
2026-09-26T12:42:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2476 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T12:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:44Z Disabled control of system clock
I20260926 12:42:44.767988 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34605
--webserver_interface=127.10.224.190
--webserver_port=0
--builtin_ntp_servers=127.10.224.148:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:42:44.883435 30847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:44.883767 30847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:44.883808 30847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:44.888160 30847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:44.888242 30847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:44.888257 30847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:44.888275 30847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:44.888291 30847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:44.893553 30847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:34605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:44.894703 30847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:44.896143 30847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:44.904783 30855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:44.904784 30852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:44.904906 30853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:44.905117 30847 server_base.cc:1034] running on GCE node
I20260926 12:42:44.905802 30847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:44.907037 30847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:44.908211 30847 hybrid_clock.cc:648] HybridClock initialized: now 1790426564908193 us; error 32 us; skew 500 ppm
I20260926 12:42:44.910183 30847 webserver.cc:460] Webserver started at http://127.10.224.190:39133/ using document root <none> and password file <none>
I20260926 12:42:44.910789 30847 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:44.910856 30847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:44.911091 30847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:44.913311 30847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "46fb7e54ab7f49ba935697a20c4595d6"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "a38f4dd266ee72c8267f2412f3622204"
server_key_iv: "2aaffd125bd8ef10fa4db2ce2e534220"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.913913 30847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "46fb7e54ab7f49ba935697a20c4595d6"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "a38f4dd266ee72c8267f2412f3622204"
server_key_iv: "2aaffd125bd8ef10fa4db2ce2e534220"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.917870 30847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 12:42:44.920506 30861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:44.921883 30847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:44.922055 30847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "46fb7e54ab7f49ba935697a20c4595d6"
format_stamp: "Formatted at 2026-09-26 12:42:44 on dist-test-slave-m3pj"
server_key: "a38f4dd266ee72c8267f2412f3622204"
server_key_iv: "2aaffd125bd8ef10fa4db2ce2e534220"
server_key_version: "encryptionkey@0"
I20260926 12:42:44.922194 30847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:44.944134 30847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:44.944922 30847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:44.945153 30847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:44.952201 30847 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:34605
I20260926 12:42:44.952212 30913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:34605 every 8 connection(s)
I20260926 12:42:44.953202 30847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:44.954424 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 30847
I20260926 12:42:44.954555 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:44.954846 11138 external_mini_cluster.cc:1428] Setting key 89a567f84cc458e20c550e38d948082e
I20260926 12:42:44.957091 30914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:44.964305 30914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6: Bootstrap starting.
I20260926 12:42:44.966619 30914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:44.967345 30914 log.cc:824] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:44.969823 30914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6: No bootstrap required, opened a new log
I20260926 12:42:44.971355 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:44.956298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.10.224.190:34605)
0926 12:42:44.956989 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:44.957008 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:44.957051 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:42:44.957911 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:44.957934 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:44.958425 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:44.958794 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:44.959249 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.960783 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.961752 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:44.962494 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:44.962667 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:44.969043 (+ 6376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:44.969051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:44.969070 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:44.969596 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:44.969677 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:44.969706 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:44.969823 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:44.969957 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 12:42:44.970257 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260926 12:42:44.974323 30914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fb7e54ab7f49ba935697a20c4595d6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34605 } }
I20260926 12:42:44.974565 30914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:44.974602 30914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fb7e54ab7f49ba935697a20c4595d6, State: Initialized, Role: FOLLOWER
I20260926 12:42:44.975036 30914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fb7e54ab7f49ba935697a20c4595d6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34605 } }
I20260926 12:42:44.975201 30914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:44.975276 30914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:44.975440 30914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:44.976461 30914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fb7e54ab7f49ba935697a20c4595d6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34605 } }
I20260926 12:42:44.976822 30914 leader_election.cc:302] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46fb7e54ab7f49ba935697a20c4595d6; no voters:
I20260926 12:42:44.977144 30914 leader_election.cc:288] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:44.977300 30919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:44.977531 30919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [term 1 LEADER]: Becoming Leader. State: Replica: 46fb7e54ab7f49ba935697a20c4595d6, State: Running, Role: LEADER
I20260926 12:42:44.977818 30919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fb7e54ab7f49ba935697a20c4595d6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34605 } }
I20260926 12:42:44.978425 30914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:44.979453 30920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46fb7e54ab7f49ba935697a20c4595d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fb7e54ab7f49ba935697a20c4595d6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34605 } } }
I20260926 12:42:44.979614 30920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:44.980008 30928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:44.980033 30921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46fb7e54ab7f49ba935697a20c4595d6. Latest consensus state: current_term: 1 leader_uuid: "46fb7e54ab7f49ba935697a20c4595d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46fb7e54ab7f49ba935697a20c4595d6" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34605 } } }
I20260926 12:42:44.980228 30921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:44.985116 30928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:44.991263 30928 catalog_manager.cc:1347] Generated new cluster ID: 619677741ecf45c7b25f92eda8b1f143
I20260926 12:42:44.991372 30928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:45.002480 30928 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:45.003747 30928 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:45.021308 30928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6: Generated new TSK 0
I20260926 12:42:45.022253 30928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:45.032305 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.129:0
--local_ip_for_outbound_sockets=127.10.224.129
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34605
--builtin_ntp_servers=127.10.224.148:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:42:45.151197 30938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:45.151588 30938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:45.151674 30938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:45.155896 30938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:45.155998 30938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:45.156096 30938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.129
I20260926 12:42:45.161198 30938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.224.129
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.129
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:45.162467 30938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:45.164026 30938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:45.174074 30946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:45.174077 30944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:45.174196 30938 server_base.cc:1034] running on GCE node
W20260926 12:42:45.174147 30943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:45.174705 30938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:45.175427 30938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:45.176625 30938 hybrid_clock.cc:648] HybridClock initialized: now 1790426565176602 us; error 32 us; skew 500 ppm
I20260926 12:42:45.178614 30938 webserver.cc:460] Webserver started at http://127.10.224.129:37483/ using document root <none> and password file <none>
I20260926 12:42:45.179303 30938 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:45.179410 30938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:45.179680 30938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:45.181845 30938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/data/instance:
uuid: "46bc0436c08d47639910facbce4c2d2e"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "2d22c9e5cdc74f428f1a609d071c0a4a"
server_key_iv: "7b4ca0a5add03dc334a8a4372430bb64"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.182463 30938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance:
uuid: "46bc0436c08d47639910facbce4c2d2e"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "2d22c9e5cdc74f428f1a609d071c0a4a"
server_key_iv: "7b4ca0a5add03dc334a8a4372430bb64"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.186324 30938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:42:45.188922 30952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.190219 30938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:42:45.190369 30938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/wal
uuid: "46bc0436c08d47639910facbce4c2d2e"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "2d22c9e5cdc74f428f1a609d071c0a4a"
server_key_iv: "7b4ca0a5add03dc334a8a4372430bb64"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.190490 30938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:45.207020 30938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:45.207878 30938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:45.208097 30938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:45.208732 30938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:45.209728 30938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:45.209800 30938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.209872 30938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:45.209923 30938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.219620 30938 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.129:44861
I20260926 12:42:45.219620 31065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.129:44861 every 8 connection(s)
I20260926 12:42:45.220592 30938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/data/info.pb
I20260926 12:42:45.229532 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 30938
I20260926 12:42:45.229646 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-0/wal/instance
I20260926 12:42:45.229978 11138 external_mini_cluster.cc:1428] Setting key 0708e3cfe7ed6568a5304ab72d362060
I20260926 12:42:45.230543 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.222634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:33883 (local address 127.10.224.190:34605)
0926 12:42:45.222782 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.222785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.223945 (+ 1160us) server_negotiation.cc:407] Connection header received
0926 12:42:45.225419 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.225422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.225481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.225571 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:45.226497 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.227058 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.228332 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.228565 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.228679 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.229974 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:45.229979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:45.229981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:45.229997 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:45.230024 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:45.230035 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:45.230117 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.230312 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.230392 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:42:45.231748 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.223046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34605 (local address 127.10.224.129:33883)
0926 12:42:45.223789 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.223831 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:45.225190 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:45.225740 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:45.225747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:45.225850 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:45.226332 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.226344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.227194 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.227200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.228197 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.228207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.228727 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.228731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.228912 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.229677 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:45.229679 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:45.229746 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:45.229819 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:45.230146 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:45.230155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:45.230161 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:42:45.230280 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 12:42:45.230602 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260926 12:42:45.232197 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.130:0
--local_ip_for_outbound_sockets=127.10.224.130
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34605
--builtin_ntp_servers=127.10.224.148:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:42:45.233114 31066 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34605
I20260926 12:42:45.233381 31066 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:45.233942 31066 heartbeater.cc:511] Master 127.10.224.190:34605 requested a full tablet report, sending...
I20260926 12:42:45.235483 30878 ts_manager.cc:194] Registered new tserver with Master: 46bc0436c08d47639910facbce4c2d2e (127.10.224.129:44861)
I20260926 12:42:45.236775 30878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.129:33883
W20260926 12:42:45.354442 31069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:45.354748 31069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:45.354789 31069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:45.359014 31069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:45.359081 31069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:45.359141 31069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.130
I20260926 12:42:45.364248 31069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.224.130
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.130
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:45.365553 31069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:45.367072 31069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:45.377563 31077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:45.377542 31075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:45.377533 31074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:45.377691 31069 server_base.cc:1034] running on GCE node
I20260926 12:42:45.378275 31069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:45.379019 31069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:45.380229 31069 hybrid_clock.cc:648] HybridClock initialized: now 1790426565380187 us; error 43 us; skew 500 ppm
I20260926 12:42:45.382547 31069 webserver.cc:460] Webserver started at http://127.10.224.130:37299/ using document root <none> and password file <none>
I20260926 12:42:45.383222 31069 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:45.383323 31069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:45.383638 31069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:45.385854 31069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/data/instance:
uuid: "cf83143080ef41c2b089206bda5f79d3"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "9fc9f207d7e077bca62c8c7f827bf073"
server_key_iv: "f30bd24c4d9e64fefb0382f649657140"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.386456 31069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance:
uuid: "cf83143080ef41c2b089206bda5f79d3"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "9fc9f207d7e077bca62c8c7f827bf073"
server_key_iv: "f30bd24c4d9e64fefb0382f649657140"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.390501 31069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 12:42:45.393198 31083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.394536 31069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:42:45.394685 31069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/wal
uuid: "cf83143080ef41c2b089206bda5f79d3"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "9fc9f207d7e077bca62c8c7f827bf073"
server_key_iv: "f30bd24c4d9e64fefb0382f649657140"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.394796 31069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:45.432070 31069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:45.432873 31069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:45.433084 31069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:45.433719 31069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:45.434738 31069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:45.434787 31069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.434828 31069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:45.435057 31069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.445246 31069 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.130:46235
I20260926 12:42:45.445269 31196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.130:46235 every 8 connection(s)
I20260926 12:42:45.446365 31069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/data/info.pb
I20260926 12:42:45.449733 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 31069
I20260926 12:42:45.449860 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/wal/instance
I20260926 12:42:45.450176 11138 external_mini_cluster.cc:1428] Setting key b5e3d82dfdca5d968c06a655a851da59
I20260926 12:42:45.452371 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.224.131:0
--local_ip_for_outbound_sockets=127.10.224.131
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34605
--builtin_ntp_servers=127.10.224.148:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:42:45.456741 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.448550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.130:55833 (local address 127.10.224.190:34605)
0926 12:42:45.448711 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.448716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.449923 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 12:42:45.451597 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.451600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.451658 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.451769 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:45.452858 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.453380 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.454568 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.454799 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.454913 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.456195 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:45.456199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:45.456202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:45.456217 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:45.456248 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:45.456256 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:45.456335 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.456512 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.456588 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:42:45.457713 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.449050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34605 (local address 127.10.224.130:55833)
0926 12:42:45.449750 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.449794 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:45.451337 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:45.451970 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:45.451978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:45.452074 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:45.452708 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.452723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.453524 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.453527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.454433 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.454444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.454970 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.454974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.455145 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.455900 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:45.455901 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:45.455951 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:45.456024 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:45.456349 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:45.456361 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:45.456369 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 12:42:45.456511 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 12:42:45.456831 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260926 12:42:45.459014 31197 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34605
I20260926 12:42:45.459291 31197 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:45.459936 31197 heartbeater.cc:511] Master 127.10.224.190:34605 requested a full tablet report, sending...
I20260926 12:42:45.461135 30878 ts_manager.cc:194] Registered new tserver with Master: cf83143080ef41c2b089206bda5f79d3 (127.10.224.130:46235)
I20260926 12:42:45.462059 30878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.130:55833
W20260926 12:42:45.570951 31200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:45.571224 31200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:45.571264 31200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:45.575393 31200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:45.575455 31200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:45.575560 31200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.224.131
I20260926 12:42:45.580883 31200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.224.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.224.131
--webserver_port=0
--tserver_master_addrs=127.10.224.190:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.224.131
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:45.582075 31200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:45.583632 31200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:45.594070 31208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:45.594137 31200 server_base.cc:1034] running on GCE node
W20260926 12:42:45.594071 31206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:45.594328 31205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:45.594785 31200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:45.595582 31200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:45.596760 31200 hybrid_clock.cc:648] HybridClock initialized: now 1790426565596721 us; error 51 us; skew 500 ppm
I20260926 12:42:45.598815 31200 webserver.cc:460] Webserver started at http://127.10.224.131:43039/ using document root <none> and password file <none>
I20260926 12:42:45.599483 31200 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:45.599610 31200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:45.599835 31200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:45.601972 31200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/data/instance:
uuid: "351080285bf64403ac38eb4aff5cb144"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "9379c31087c1d38447413f40e5b063de"
server_key_iv: "4e57109984700c71ae2e14b99d01b980"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.602604 31200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance:
uuid: "351080285bf64403ac38eb4aff5cb144"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "9379c31087c1d38447413f40e5b063de"
server_key_iv: "4e57109984700c71ae2e14b99d01b980"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.606760 31200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:42:45.609606 31214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.611052 31200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 12:42:45.611223 31200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/wal
uuid: "351080285bf64403ac38eb4aff5cb144"
format_stamp: "Formatted at 2026-09-26 12:42:45 on dist-test-slave-m3pj"
server_key: "9379c31087c1d38447413f40e5b063de"
server_key_iv: "4e57109984700c71ae2e14b99d01b980"
server_key_version: "encryptionkey@0"
I20260926 12:42:45.611346 31200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:45.630570 31200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:45.631342 31200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:45.631623 31200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:45.632269 31200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:42:45.633273 31200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:42:45.633347 31200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.633411 31200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:42:45.633462 31200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:45.642954 31200 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.131:34799
I20260926 12:42:45.642975 31327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.131:34799 every 8 connection(s)
I20260926 12:42:45.644114 31200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/data/info.pb
I20260926 12:42:45.649261 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 31200
I20260926 12:42:45.649427 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/wal/instance
I20260926 12:42:45.649708 11138 external_mini_cluster.cc:1428] Setting key b953e93aadebf9ae6d6b156acf9a49f4
I20260926 12:42:45.654582 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.646303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.131:44113 (local address 127.10.224.190:34605)
0926 12:42:45.646475 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.646479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.647585 (+ 1106us) server_negotiation.cc:407] Connection header received
0926 12:42:45.649016 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.649020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.649087 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.649190 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:45.650277 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.651020 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.652369 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.652597 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.652683 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.654065 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:45.654068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:45.654071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:45.654084 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:45.654106 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:45.654114 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:45.654183 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.654357 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.654456 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 12:42:45.655496 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.646678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34605 (local address 127.10.224.131:44113)
0926 12:42:45.647364 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.647405 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:45.648778 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:45.649349 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:45.649356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:45.649535 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:42:45.650105 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.650119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.651152 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.651157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.652259 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.652269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.652769 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.652773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.653009 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.653774 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:42:45.653775 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:42:45.653820 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:42:45.653895 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:42:45.654194 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:42:45.654201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:42:45.654205 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:42:45.654306 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 12:42:45.654582 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260926 12:42:45.656872 31328 heartbeater.cc:347] Connected to a master server at 127.10.224.190:34605
I20260926 12:42:45.657164 31328 heartbeater.cc:464] Registering TS with master...
I20260926 12:42:45.657780 31328 heartbeater.cc:511] Master 127.10.224.190:34605 requested a full tablet report, sending...
I20260926 12:42:45.659035 30878 ts_manager.cc:194] Registered new tserver with Master: 351080285bf64403ac38eb4aff5cb144 (127.10.224.131:34799)
I20260926 12:42:45.659783 30878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.224.131:44113
I20260926 12:42:45.664281 11138 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:42:45.670622 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.666413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.10.224.190:34605)
0926 12:42:45.666588 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.666593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.666730 (+ 137us) server_negotiation.cc:407] Connection header received
0926 12:42:45.666932 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.666935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.666983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.667075 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:45.667370 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.667986 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.668914 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.669150 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.669262 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.670151 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:45.670154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:45.670156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:45.670169 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:45.670188 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:45.670196 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:45.670275 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.670407 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.670468 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 12:42:45.674070 30878 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:42:45.677608 30878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:42:45.694239 31342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.687727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.10.224.129:44861)
0926 12:42:45.688194 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.688206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.688669 (+ 463us) server_negotiation.cc:407] Connection header received
0926 12:42:45.688892 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.688898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.689119 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.689249 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:45.689860 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.690968 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.692625 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.693355 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.693501 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.693714 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.693932 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.694070 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":114,"threads_started":1}
I20260926 12:42:45.694233 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.687330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44861 (local address 127.0.0.1:39558)
0926 12:42:45.688571 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.688590 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:45.688724 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:45.689456 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:45.689463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:45.689495 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:45.689715 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.689723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.691249 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.691252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.692491 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.692501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.693506 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.693510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.693667 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.693708 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 12:42:45.693864 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 12:42:45.694004 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"mutex_wait_us":93,"spinlock_wait_cycles":14720,"thread_start_us":111,"threads_started":1}
I20260926 12:42:45.695626 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.687308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:34799 (local address 127.0.0.1:47994)
0926 12:42:45.688269 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.688490 (+ 221us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:45.688706 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:45.689456 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:45.689463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:45.689497 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:45.689715 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.689723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.692463 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.692467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.693699 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.693710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.695011 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.695014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.695203 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.695257 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 12:42:45.695354 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 12:42:45.695440 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":97,"spinlock_wait_cycles":17536,"thread_start_us":120,"threads_started":1}
I20260926 12:42:45.695832 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.687444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.10.224.131:34799)
0926 12:42:45.688396 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.688409 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.688585 (+ 176us) server_negotiation.cc:407] Connection header received
0926 12:42:45.688892 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.688898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.689119 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.689248 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:45.691117 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.692336 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.693809 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.694886 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.695023 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.695203 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.695423 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.695552 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":173,"threads_started":1}
I20260926 12:42:45.696660 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.688199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:46235 (local address 127.0.0.1:35256)
0926 12:42:45.688978 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.688992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:45.689081 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:45.689494 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:45.689498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:45.689531 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:45.689726 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.689733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.692784 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.692788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.693940 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.693949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.696253 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.696257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.696344 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.696367 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:42:45.696442 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 12:42:45.696513 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":33,"spinlock_wait_cycles":9344,"thread_start_us":92,"threads_started":1}
I20260926 12:42:45.696702 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.688614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.10.224.130:46235)
0926 12:42:45.689038 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.689051 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.689075 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:42:45.689196 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.689201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.689354 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.689511 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:45.691618 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.692671 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.694125 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.696119 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.696252 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.696405 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.696456 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.696544 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":104,"threads_started":1}
I20260926 12:42:45.697777 31000 tablet_service.cc:1467] Processing CreateTablet for tablet ec25b4e676134150a39e81bbf6b823c3 (DEFAULT_TABLE table=test-table [id=a0b4f429ae0c4b51a5e25e1f1f2fdb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:45.699029 31000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec25b4e676134150a39e81bbf6b823c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:45.698740 31262 tablet_service.cc:1467] Processing CreateTablet for tablet ec25b4e676134150a39e81bbf6b823c3 (DEFAULT_TABLE table=test-table [id=a0b4f429ae0c4b51a5e25e1f1f2fdb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:45.699816 31262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec25b4e676134150a39e81bbf6b823c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:45.699873 31131 tablet_service.cc:1467] Processing CreateTablet for tablet ec25b4e676134150a39e81bbf6b823c3 (DEFAULT_TABLE table=test-table [id=a0b4f429ae0c4b51a5e25e1f1f2fdb58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:42:45.700856 31131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec25b4e676134150a39e81bbf6b823c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:45.706351 31345 tablet_bootstrap.cc:492] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: Bootstrap starting.
I20260926 12:42:45.708122 31347 tablet_bootstrap.cc:492] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: Bootstrap starting.
I20260926 12:42:45.708714 31346 tablet_bootstrap.cc:492] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e: Bootstrap starting.
I20260926 12:42:45.709066 31345 tablet_bootstrap.cc:654] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:45.709956 31345 log.cc:824] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:45.710649 31347 tablet_bootstrap.cc:654] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:45.710733 31346 tablet_bootstrap.cc:654] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:45.711575 31347 log.cc:824] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:45.711575 31346 log.cc:824] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:45.712311 31345 tablet_bootstrap.cc:492] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: No bootstrap required, opened a new log
I20260926 12:42:45.712553 31345 ts_tablet_manager.cc:1397] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:42:45.713425 31346 tablet_bootstrap.cc:492] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e: No bootstrap required, opened a new log
I20260926 12:42:45.713629 31346 ts_tablet_manager.cc:1397] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:42:45.716365 31345 raft_consensus.cc:348] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.716619 31345 raft_consensus.cc:374] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:45.716687 31345 raft_consensus.cc:729] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351080285bf64403ac38eb4aff5cb144, State: Initialized, Role: FOLLOWER
I20260926 12:42:45.717108 31345 consensus_queue.cc:260] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.717535 31346 raft_consensus.cc:348] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.717795 31346 raft_consensus.cc:374] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:45.717828 31328 heartbeater.cc:503] Master 127.10.224.190:34605 was elected leader, sending a full tablet report...
I20260926 12:42:45.717859 31346 raft_consensus.cc:729] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46bc0436c08d47639910facbce4c2d2e, State: Initialized, Role: FOLLOWER
I20260926 12:42:45.718128 31345 ts_tablet_manager.cc:1428] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:42:45.718312 31346 consensus_queue.cc:260] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.719022 31066 heartbeater.cc:503] Master 127.10.224.190:34605 was elected leader, sending a full tablet report...
I20260926 12:42:45.719261 31346 ts_tablet_manager.cc:1428] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:42:45.721263 31347 tablet_bootstrap.cc:492] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: No bootstrap required, opened a new log
I20260926 12:42:45.721506 31347 ts_tablet_manager.cc:1397] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
W20260926 12:42:45.721784 31067 tablet.cc:2369] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:42:45.725699 31347 raft_consensus.cc:348] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.726016 31347 raft_consensus.cc:374] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:45.726082 31347 raft_consensus.cc:729] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf83143080ef41c2b089206bda5f79d3, State: Initialized, Role: FOLLOWER
I20260926 12:42:45.726547 31347 consensus_queue.cc:260] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.727331 31197 heartbeater.cc:503] Master 127.10.224.190:34605 was elected leader, sending a full tablet report...
I20260926 12:42:45.727643 31347 ts_tablet_manager.cc:1428] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260926 12:42:45.729583 31352 raft_consensus.cc:482] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:42:45.729907 31352 raft_consensus.cc:504] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.731156 31352 leader_election.cc:288] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 351080285bf64403ac38eb4aff5cb144 (127.10.224.131:34799), cf83143080ef41c2b089206bda5f79d3 (127.10.224.130:46235)
I20260926 12:42:45.735052 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.731401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.131:34799 (local address 127.10.224.129:33629)
0926 12:42:45.731557 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.731575 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:45.731667 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:45.732270 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:45.732272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:45.732289 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:45.732380 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.732385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.733173 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.733176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.733981 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.733990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.734617 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.734621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.734721 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.734750 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:42:45.734833 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:42:45.734911 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 12:42:45.735052 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.731497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:33629 (local address 127.10.224.131:34799)
0926 12:42:45.731854 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.731861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.731878 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:42:45.731947 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.731950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.732005 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.732082 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:45.732498 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.733034 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.734106 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.734501 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.734629 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.734686 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.734849 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.734921 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260926 12:42:45.735268 31354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.731721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.130:46235 (local address 127.10.224.129:49283)
0926 12:42:45.732143 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.732157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:42:45.732263 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:42:45.732525 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:42:45.732527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:42:45.732543 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:42:45.732671 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.732676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.733409 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.733412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.734183 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:42:45.734191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.734843 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.734847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:42:45.734951 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.734969 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:42:45.735053 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:42:45.735126 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":130,"threads_started":1}
I20260926 12:42:45.735256 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.732070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.224.129:49283 (local address 127.10.224.130:46235)
0926 12:42:45.732186 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.732192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.732211 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:42:45.732369 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.732372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.732420 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.732503 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:42:45.732781 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.733277 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.734314 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.734704 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.734843 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.734890 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.735081 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.735153 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 12:42:45.736217 31282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec25b4e676134150a39e81bbf6b823c3" candidate_uuid: "46bc0436c08d47639910facbce4c2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351080285bf64403ac38eb4aff5cb144" is_pre_election: true
I20260926 12:42:45.736210 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec25b4e676134150a39e81bbf6b823c3" candidate_uuid: "46bc0436c08d47639910facbce4c2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf83143080ef41c2b089206bda5f79d3" is_pre_election: true
I20260926 12:42:45.736547 31282 raft_consensus.cc:2454] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46bc0436c08d47639910facbce4c2d2e in term 0.
I20260926 12:42:45.736547 31151 raft_consensus.cc:2454] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46bc0436c08d47639910facbce4c2d2e in term 0.
I20260926 12:42:45.737119 30956 leader_election.cc:302] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46bc0436c08d47639910facbce4c2d2e, cf83143080ef41c2b089206bda5f79d3; no voters:
I20260926 12:42:45.737392 31352 raft_consensus.cc:2781] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:42:45.737480 31352 raft_consensus.cc:482] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:42:45.737512 31352 raft_consensus.cc:3037] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:45.738612 31352 raft_consensus.cc:504] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.738979 31352 leader_election.cc:288] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [CANDIDATE]: Term 1 election: Requested vote from peers 351080285bf64403ac38eb4aff5cb144 (127.10.224.131:34799), cf83143080ef41c2b089206bda5f79d3 (127.10.224.130:46235)
I20260926 12:42:45.739363 31282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec25b4e676134150a39e81bbf6b823c3" candidate_uuid: "46bc0436c08d47639910facbce4c2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351080285bf64403ac38eb4aff5cb144"
I20260926 12:42:45.739454 31151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec25b4e676134150a39e81bbf6b823c3" candidate_uuid: "46bc0436c08d47639910facbce4c2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf83143080ef41c2b089206bda5f79d3"
I20260926 12:42:45.739501 31282 raft_consensus.cc:3037] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:45.739636 31151 raft_consensus.cc:3037] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:45.740701 31151 raft_consensus.cc:2454] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46bc0436c08d47639910facbce4c2d2e in term 1.
I20260926 12:42:45.740715 31282 raft_consensus.cc:2454] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46bc0436c08d47639910facbce4c2d2e in term 1.
I20260926 12:42:45.741047 30956 leader_election.cc:302] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351080285bf64403ac38eb4aff5cb144, 46bc0436c08d47639910facbce4c2d2e; no voters:
I20260926 12:42:45.741242 31352 raft_consensus.cc:2781] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:45.741484 31352 raft_consensus.cc:686] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [term 1 LEADER]: Becoming Leader. State: Replica: 46bc0436c08d47639910facbce4c2d2e, State: Running, Role: LEADER
I20260926 12:42:45.741906 31352 consensus_queue.cc:237] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } }
I20260926 12:42:45.745600 30877 catalog_manager.cc:5495] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 46bc0436c08d47639910facbce4c2d2e (127.10.224.129). New cstate: current_term: 1 leader_uuid: "46bc0436c08d47639910facbce4c2d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46bc0436c08d47639910facbce4c2d2e" member_type: VOTER last_known_addr { host: "127.10.224.129" port: 44861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 } health_report { overall_health: UNKNOWN } } }
I20260926 12:42:45.764750 31342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.760888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.10.224.129:44861)
0926 12:42:45.761064 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:45.761071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:45.761087 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:42:45.761173 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:45.761175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:45.761231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:45.761346 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:42:45.761593 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.762110 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.763050 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:45.763277 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:45.763390 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:45.763797 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:42:45.764271 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:42:45.764420 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:45.764475 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 12:42:45.764538 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 12:42:45.771330 31282 raft_consensus.cc:1261] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 1 FOLLOWER]: Refusing update from remote peer 46bc0436c08d47639910facbce4c2d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:45.771330 31151 raft_consensus.cc:1261] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 1 FOLLOWER]: Refusing update from remote peer 46bc0436c08d47639910facbce4c2d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:42:45.772128 31355 consensus_queue.cc:1035] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "351080285bf64403ac38eb4aff5cb144" member_type: VOTER last_known_addr { host: "127.10.224.131" port: 34799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:45.772321 31352 consensus_queue.cc:1035] T ec25b4e676134150a39e81bbf6b823c3 P 46bc0436c08d47639910facbce4c2d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf83143080ef41c2b089206bda5f79d3" member_type: VOTER last_known_addr { host: "127.10.224.130" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:42:45.781020 31360 mvcc.cc:205] Tried to move back new op lower bound from 7333587213391212544 to 7333587213282390016. Current Snapshot: MvccSnapshot[applied={T|T < 7333587213391212544}]
I20260926 12:42:45.782012 31359 mvcc.cc:205] Tried to move back new op lower bound from 7333587213391212544 to 7333587213282390016. Current Snapshot: MvccSnapshot[applied={T|T < 7333587213391212544}]
I20260926 12:42:45.790159 30876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:45.790330 30876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:42:45.792990 30876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46fb7e54ab7f49ba935697a20c4595d6: Sending DeleteTablet for 3 replicas of tablet ec25b4e676134150a39e81bbf6b823c3
I20260926 12:42:45.793831 31000 tablet_service.cc:1514] Processing DeleteTablet for tablet ec25b4e676134150a39e81bbf6b823c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:45 UTC) from {username='slave'} at 127.0.0.1:39558
I20260926 12:42:45.794130 31131 tablet_service.cc:1514] Processing DeleteTablet for tablet ec25b4e676134150a39e81bbf6b823c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:45 UTC) from {username='slave'} at 127.0.0.1:35256
I20260926 12:42:45.794430 31262 tablet_service.cc:1514] Processing DeleteTablet for tablet ec25b4e676134150a39e81bbf6b823c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:42:45 UTC) from {username='slave'} at 127.0.0.1:47994
I20260926 12:42:45.794647 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 30938
I20260926 12:42:45.795476 31370 tablet_replica.cc:330] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: stopping tablet replica
I20260926 12:42:45.795650 31369 tablet_replica.cc:330] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: stopping tablet replica
I20260926 12:42:45.795995 31369 raft_consensus.cc:2229] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:45.795995 31370 raft_consensus.cc:2229] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:42:45.796363 31370 raft_consensus.cc:2258] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:45.796363 31369 raft_consensus.cc:2258] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:42:45.798105 31369 ts_tablet_manager.cc:1905] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:45.798441 31370 ts_tablet_manager.cc:1905] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:42:45.801059 31369 ts_tablet_manager.cc:1918] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:45.801188 31369 log.cc:1196] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-1/wal/wals/ec25b4e676134150a39e81bbf6b823c3
I20260926 12:42:45.801514 31369 ts_tablet_manager.cc:1939] T ec25b4e676134150a39e81bbf6b823c3 P cf83143080ef41c2b089206bda5f79d3: Deleting consensus metadata
I20260926 12:42:45.801985 31370 ts_tablet_manager.cc:1918] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:42:45.802094 31370 log.cc:1196] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: Deleting WAL directory at /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790426298078858-11138-0/minicluster-data/ts-2/wal/wals/ec25b4e676134150a39e81bbf6b823c3
I20260926 12:42:45.802446 31370 ts_tablet_manager.cc:1939] T ec25b4e676134150a39e81bbf6b823c3 P 351080285bf64403ac38eb4aff5cb144: Deleting consensus metadata
I20260926 12:42:45.802949 30865 catalog_manager.cc:4841] TS cf83143080ef41c2b089206bda5f79d3 (127.10.224.130:46235): tablet ec25b4e676134150a39e81bbf6b823c3 (table test-table [id=a0b4f429ae0c4b51a5e25e1f1f2fdb58]) successfully deleted
W20260926 12:42:45.803191 30863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.224.129:44861 (error 108)
I20260926 12:42:45.803766 30865 catalog_manager.cc:4841] TS 351080285bf64403ac38eb4aff5cb144 (127.10.224.131:34799): tablet ec25b4e676134150a39e81bbf6b823c3 (table test-table [id=a0b4f429ae0c4b51a5e25e1f1f2fdb58]) successfully deleted
I20260926 12:42:45.804431 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:45.804095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.129:44861 (local address 127.0.0.1:39576)
0926 12:42:45.804236 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 12:42:45.804319 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.129:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 12:42:45.804486 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 31069
W20260926 12:42:45.804623 30863 catalog_manager.cc:4584] TS 46bc0436c08d47639910facbce4c2d2e (127.10.224.129:44861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec25b4e676134150a39e81bbf6b823c3: Network error: Client connection negotiation failed: client connection to 127.10.224.129:44861: connect: Connection refused (error 111)
I20260926 12:42:45.810388 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 31200
I20260926 12:42:45.816102 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 30847
2026-09-26T12:42:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1079 ms)
[----------] 6 tests from EncryptionPolicyTest (10934 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:45 dist-test-slave-m3pj krb5kdc[31374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:45 dist-test-slave-m3pj krb5kdc[31374](info): set up 2 sockets
Sep 26 12:42:45 dist-test-slave-m3pj krb5kdc[31374](info): commencing operation
krb5kdc: starting...
W20260926 12:42:46.880329 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:46 dist-test-slave-m3pj krb5kdc[31374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:47.015439 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:47.131863 31387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:47.132210 31387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:47.132293 31387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:47.136493 31387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:47.136605 31387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:47.136643 31387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:47.136698 31387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:47.136734 31387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:47.142326 31387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:47.143569 31387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:47.144685 31387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:47.153805 31393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:47.153802 31395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:47.153802 31392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:47.153848 31387 server_base.cc:1034] running on GCE node
I20260926 12:42:47.154816 31387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:47.156100 31387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:47.157297 31387 hybrid_clock.cc:648] HybridClock initialized: now 1790426567157282 us; error 30 us; skew 500 ppm
Sep 26 12:42:47 dist-test-slave-m3pj krb5kdc[31374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:47.160256 31387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:47.161248 31387 webserver.cc:460] Webserver started at http://127.0.0.1:43475/ using document root <none> and password file <none>
I20260926 12:42:47.161796 31387 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:47.161857 31387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:47.162031 31387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:47.164289 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "b4fedd50b5974186b728b52ba87519d1"
format_stamp: "Formatted at 2026-09-26 12:42:47 on dist-test-slave-m3pj"
server_key: "e0106fa5fbcd9bdb5ae7bb7e5b561697"
server_key_iv: "ea36d16b05f8004c7091eacb436b493b"
server_key_version: "encryptionkey@0"
I20260926 12:42:47.164870 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "b4fedd50b5974186b728b52ba87519d1"
format_stamp: "Formatted at 2026-09-26 12:42:47 on dist-test-slave-m3pj"
server_key: "e0106fa5fbcd9bdb5ae7bb7e5b561697"
server_key_iv: "ea36d16b05f8004c7091eacb436b493b"
server_key_version: "encryptionkey@0"
I20260926 12:42:47.168941 31387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:47.171653 31402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:47.173066 31387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:47.173210 31387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "b4fedd50b5974186b728b52ba87519d1"
format_stamp: "Formatted at 2026-09-26 12:42:47 on dist-test-slave-m3pj"
server_key: "e0106fa5fbcd9bdb5ae7bb7e5b561697"
server_key_iv: "ea36d16b05f8004c7091eacb436b493b"
server_key_version: "encryptionkey@0"
I20260926 12:42:47.173336 31387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:47.187861 31387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:47.191321 31387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:47.191568 31387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:47.198995 31387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35839
I20260926 12:42:47.199021 31454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35839 every 8 connection(s)
I20260926 12:42:47.200093 31387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:47.202920 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 31387
I20260926 12:42:47.203090 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:47.203088 31455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:47.203393 11138 external_mini_cluster.cc:1428] Setting key ca3a458fd1e7b1f170cd9154717c3cbd
I20260926 12:42:47.210989 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1: Bootstrap starting.
I20260926 12:42:47.213652 31455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1: Neither blocks nor log segments found. Creating new log.
Sep 26 12:42:47 dist-test-slave-m3pj krb5kdc[31374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:42:47.214529 31455 log.cc:824] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:47.217157 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1: No bootstrap required, opened a new log
I20260926 12:42:47.220526 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:47.204984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52907 (local address 127.0.0.1:35839)
0926 12:42:47.205481 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:47.205502 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:47.205548 (+ 46us) server_negotiation.cc:407] Connection header received
0926 12:42:47.206766 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:47.206796 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:47.207281 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:47.207792 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:47.208541 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:47.210073 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:47.210924 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:47.211758 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:47.211917 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:47.214602 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:47.214611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:47.214631 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:47.214667 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:47.216893 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:47.217569 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:47.217576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:47.217582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:47.217723 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:47.218044 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:47.218047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:47.218049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:47.218579 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:47.218784 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:47.219101 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 12:42:47.219360 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":141,"threads_started":1}
I20260926 12:42:47.222078 31455 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fedd50b5974186b728b52ba87519d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35839 } }
I20260926 12:42:47.222361 31455 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:47.222441 31455 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fedd50b5974186b728b52ba87519d1, State: Initialized, Role: FOLLOWER
I20260926 12:42:47.222955 31455 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fedd50b5974186b728b52ba87519d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35839 } }
I20260926 12:42:47.223124 31455 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:47.223233 31455 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:47.223382 31455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:47.223959 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:42:47.224526 31455 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fedd50b5974186b728b52ba87519d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35839 } }
I20260926 12:42:47.224889 31455 leader_election.cc:302] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4fedd50b5974186b728b52ba87519d1; no voters:
I20260926 12:42:47.225237 31455 leader_election.cc:288] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:47.225667 31465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:47.225873 31465 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [term 1 LEADER]: Becoming Leader. State: Replica: b4fedd50b5974186b728b52ba87519d1, State: Running, Role: LEADER
I20260926 12:42:47.226325 31465 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fedd50b5974186b728b52ba87519d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35839 } }
I20260926 12:42:47.226732 31455 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:47.228765 31462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4fedd50b5974186b728b52ba87519d1. Latest consensus state: current_term: 1 leader_uuid: "b4fedd50b5974186b728b52ba87519d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fedd50b5974186b728b52ba87519d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35839 } } }
I20260926 12:42:47.228762 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4fedd50b5974186b728b52ba87519d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4fedd50b5974186b728b52ba87519d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35839 } } }
I20260926 12:42:47.228919 31462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:47.228924 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1 [sys.catalog]: This master's current role is: LEADER
W20260926 12:42:47.232440 31479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:42:47.232566 31479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:42:47.232654 31480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:47.233609 31480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:47.236548 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:47.226288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37183 (local address 127.0.0.1:35839)
0926 12:42:47.226411 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:47.226415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:47.226691 (+ 276us) server_negotiation.cc:407] Connection header received
0926 12:42:47.226806 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:47.226810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:47.226865 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:47.226948 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:47.228068 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:47.228894 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:47.230088 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:47.230434 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:47.230543 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:47.232528 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:47.232534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:47.232537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:47.232566 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:47.234335 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:47.234916 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:47.234920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:47.234922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:47.235016 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:47.235468 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:47.235473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:47.235476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:47.235802 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:47.235953 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:47.236267 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 12:42:47.236372 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:42:47.239981 31480 catalog_manager.cc:1347] Generated new cluster ID: 862f2298ba0c41d08c17542fdd3ef9c1
I20260926 12:42:47.240057 31480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:47.250381 31480 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:47.251840 31480 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:47.258328 31480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4fedd50b5974186b728b52ba87519d1: Generated new TSK 0
I20260926 12:42:47.259212 31480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:47.266165 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 31387
W20260926 12:42:47.271152 31372 connection.cc:535] client connection to 127.0.0.1:35839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35839): Connection reset by peer (error 104)
2026-09-26T12:42:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:47 dist-test-slave-m3pj krb5kdc[31486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:47 dist-test-slave-m3pj krb5kdc[31486](info): set up 2 sockets
Sep 26 12:42:47 dist-test-slave-m3pj krb5kdc[31486](info): commencing operation
krb5kdc: starting...
W20260926 12:42:48.320345 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:48 dist-test-slave-m3pj krb5kdc[31486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:48.451052 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:48.567112 31499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:48.567482 31499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:48.567592 31499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:48.571765 31499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:48.571887 31499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:48.571928 31499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:48.571988 31499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:48.572028 31499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:48.577618 31499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:48.578855 31499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:48.580019 31499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:48.589279 31504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:48.589315 31505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:48.589279 31507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:48.589594 31499 server_base.cc:1034] running on GCE node
I20260926 12:42:48.590291 31499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:48.591594 31499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:48.592787 31499 hybrid_clock.cc:648] HybridClock initialized: now 1790426568592754 us; error 58 us; skew 500 ppm
Sep 26 12:42:48 dist-test-slave-m3pj krb5kdc[31486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:48.595747 31499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:48.596658 31499 webserver.cc:460] Webserver started at http://127.0.0.1:39171/ using document root <none> and password file <none>
I20260926 12:42:48.597210 31499 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:48.597271 31499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:48.597438 31499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:48.599597 31499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "44bbc14112a242d6ba0db44e9ca62e68"
format_stamp: "Formatted at 2026-09-26 12:42:48 on dist-test-slave-m3pj"
server_key: "cee0f09a9fcafa220d03767f98a84476"
server_key_iv: "81cd7a81ca7ed63115b49e3b1da68f1b"
server_key_version: "encryptionkey@0"
I20260926 12:42:48.600174 31499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "44bbc14112a242d6ba0db44e9ca62e68"
format_stamp: "Formatted at 2026-09-26 12:42:48 on dist-test-slave-m3pj"
server_key: "cee0f09a9fcafa220d03767f98a84476"
server_key_iv: "81cd7a81ca7ed63115b49e3b1da68f1b"
server_key_version: "encryptionkey@0"
I20260926 12:42:48.604084 31499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:48.606865 31514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:48.608538 31499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:48.608672 31499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "44bbc14112a242d6ba0db44e9ca62e68"
format_stamp: "Formatted at 2026-09-26 12:42:48 on dist-test-slave-m3pj"
server_key: "cee0f09a9fcafa220d03767f98a84476"
server_key_iv: "81cd7a81ca7ed63115b49e3b1da68f1b"
server_key_version: "encryptionkey@0"
I20260926 12:42:48.608788 31499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:48.649710 31499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:48.653641 31499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:48.653813 31499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:48.661408 31499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36627
I20260926 12:42:48.661428 31566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36627 every 8 connection(s)
I20260926 12:42:48.662492 31499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:48.665550 31567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:48.669273 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 31499
I20260926 12:42:48.669448 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:48.669756 11138 external_mini_cluster.cc:1428] Setting key e4cadab0b5e0d00827295c55b2826e5c
I20260926 12:42:48.672216 31567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68: Bootstrap starting.
I20260926 12:42:48.674513 31567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:48.675292 31567 log.cc:824] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:48.677472 31567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68: No bootstrap required, opened a new log
I20260926 12:42:48.679692 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:48.671291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35911 (local address 127.0.0.1:36627)
0926 12:42:48.671780 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:48.671795 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:48.671833 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:42:48.672561 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:48.672580 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:48.672948 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:48.673297 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:48.673786 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:48.674853 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:48.675897 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:48.676384 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:48.676566 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:48.677617 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:48.677624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:48.677641 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:48.678130 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:48.678185 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:48.678208 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:48.678360 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:48.678544 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 12:42:48.678693 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":165,"threads_started":1}
I20260926 12:42:48.680992 31567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bbc14112a242d6ba0db44e9ca62e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260926 12:42:48.681237 31567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:48.681315 31567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44bbc14112a242d6ba0db44e9ca62e68, State: Initialized, Role: FOLLOWER
I20260926 12:42:48.681824 31567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bbc14112a242d6ba0db44e9ca62e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260926 12:42:48.681993 31567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:48.682088 31567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:48.682192 31567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:48.682850 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:42:48.683305 31567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bbc14112a242d6ba0db44e9ca62e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260926 12:42:48.683706 31567 leader_election.cc:302] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44bbc14112a242d6ba0db44e9ca62e68; no voters:
I20260926 12:42:48.684165 31567 leader_election.cc:288] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:48.684242 31572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:48.684495 31572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [term 1 LEADER]: Becoming Leader. State: Replica: 44bbc14112a242d6ba0db44e9ca62e68, State: Running, Role: LEADER
I20260926 12:42:48.684880 31572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bbc14112a242d6ba0db44e9ca62e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260926 12:42:48.685621 31567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:48.686137 31572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44bbc14112a242d6ba0db44e9ca62e68. Latest consensus state: current_term: 1 leader_uuid: "44bbc14112a242d6ba0db44e9ca62e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bbc14112a242d6ba0db44e9ca62e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } } }
I20260926 12:42:48.686321 31572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:48.686693 31577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44bbc14112a242d6ba0db44e9ca62e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44bbc14112a242d6ba0db44e9ca62e68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } } }
I20260926 12:42:48.686857 31577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:48.686918 31583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:48.690497 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:48.685540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50929 (local address 127.0.0.1:36627)
0926 12:42:48.685661 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:48.685665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:48.685677 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:42:48.685720 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:48.685723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:48.685771 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:48.685872 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:48.686208 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:48.687095 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:48.688338 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:48.688665 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:48.688783 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:48.689766 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:48.689769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:48.689772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:48.689786 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:48.689815 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:48.689829 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:48.689904 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:48.690318 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 12:42:48.690354 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:42:48.690804 31583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:48.699820 31583 catalog_manager.cc:1347] Generated new cluster ID: 80d1f0460900474ab536fba7b037aa31
I20260926 12:42:48.699913 31583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:48.720103 31583 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:48.721426 31583 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:48.727933 31583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44bbc14112a242d6ba0db44e9ca62e68: Generated new TSK 0
I20260926 12:42:48.728931 31583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:48.736218 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 31499
W20260926 12:42:48.741859 31484 connection.cc:535] client connection to 127.0.0.1:36627 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36627): Connection reset by peer (error 104)
2026-09-26T12:42:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:48 dist-test-slave-m3pj krb5kdc[31598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:48 dist-test-slave-m3pj krb5kdc[31598](info): set up 2 sockets
Sep 26 12:42:48 dist-test-slave-m3pj krb5kdc[31598](info): commencing operation
krb5kdc: starting...
W20260926 12:42:49.792238 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:49 dist-test-slave-m3pj krb5kdc[31598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:49.918991 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:50.032912 31611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:50.033237 31611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:50.033339 31611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:50.037469 31611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:50.037564 31611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:50.037603 31611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:50.037663 31611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:50.037704 31611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:50.043309 31611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33363
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:50.044524 31611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:50.045586 31611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:50.054461 31619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:50.054452 31617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:50.054456 31616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:50.054698 31611 server_base.cc:1034] running on GCE node
I20260926 12:42:50.055467 31611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:50.056636 31611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:50.057813 31611 hybrid_clock.cc:648] HybridClock initialized: now 1790426570057788 us; error 35 us; skew 500 ppm
Sep 26 12:42:50 dist-test-slave-m3pj krb5kdc[31598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:50.067109 31611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:50.068143 31611 webserver.cc:460] Webserver started at http://127.0.0.1:34891/ using document root <none> and password file <none>
I20260926 12:42:50.068710 31611 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:50.068773 31611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:50.068954 31611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:50.071043 31611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "8a821abc91dc4e67b726baa767314a2c"
format_stamp: "Formatted at 2026-09-26 12:42:50 on dist-test-slave-m3pj"
server_key: "72d57372d1644873ff414991aa84fcb2"
server_key_iv: "5b212bada309c82b32c9b3121c44a352"
server_key_version: "encryptionkey@0"
I20260926 12:42:50.071631 31611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "8a821abc91dc4e67b726baa767314a2c"
format_stamp: "Formatted at 2026-09-26 12:42:50 on dist-test-slave-m3pj"
server_key: "72d57372d1644873ff414991aa84fcb2"
server_key_iv: "5b212bada309c82b32c9b3121c44a352"
server_key_version: "encryptionkey@0"
I20260926 12:42:50.075457 31611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:42:50.077913 31626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:50.079288 31611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:42:50.079463 31611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "8a821abc91dc4e67b726baa767314a2c"
format_stamp: "Formatted at 2026-09-26 12:42:50 on dist-test-slave-m3pj"
server_key: "72d57372d1644873ff414991aa84fcb2"
server_key_iv: "5b212bada309c82b32c9b3121c44a352"
server_key_version: "encryptionkey@0"
I20260926 12:42:50.079665 31611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:50.084676 31611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:50.088454 31611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:50.088716 31611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:50.096055 31611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33363
I20260926 12:42:50.096119 31678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33363 every 8 connection(s)
I20260926 12:42:50.097034 31611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:50.100070 31679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:50.105878 31679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c: Bootstrap starting.
I20260926 12:42:50.106087 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 31611
I20260926 12:42:50.106241 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:50.106544 11138 external_mini_cluster.cc:1428] Setting key 58ff5958fb4e6259d56b63bb80aed698
I20260926 12:42:50.108848 31679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:50.109771 31679 log.cc:824] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:50.112416 31679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c: No bootstrap required, opened a new log
I20260926 12:42:50.112741 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:50.107974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37707 (local address 127.0.0.1:33363)
0926 12:42:50.108419 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:50.108436 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:50.108478 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:42:50.109417 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:50.109440 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:50.109826 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:50.110239 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:50.111170 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:50.111175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:50.111187 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:50.111200 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:50.111257 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:50.111277 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:50.111394 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:50.111697 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 12:42:50.111923 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":108,"threads_started":1}
I20260926 12:42:50.115998 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:42:50.116914 31679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a821abc91dc4e67b726baa767314a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33363 } }
I20260926 12:42:50.117189 31679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:50.117271 31679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a821abc91dc4e67b726baa767314a2c, State: Initialized, Role: FOLLOWER
I20260926 12:42:50.117781 31679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a821abc91dc4e67b726baa767314a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33363 } }
I20260926 12:42:50.117949 31679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:50.118052 31679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:50.118157 31679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:50.119201 31679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a821abc91dc4e67b726baa767314a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33363 } }
I20260926 12:42:50.119642 31679 leader_election.cc:302] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a821abc91dc4e67b726baa767314a2c; no voters:
I20260926 12:42:50.120038 31679 leader_election.cc:288] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:50.120114 31689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:50.120430 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:50.118376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36567 (local address 127.0.0.1:33363)
0926 12:42:50.118526 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:50.118530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:50.118756 (+ 226us) server_negotiation.cc:407] Connection header received
0926 12:42:50.118917 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:50.118920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:50.118959 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:50.119065 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:50.119886 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:50.119890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:50.119893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:50.119913 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:50.119949 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:50.119961 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:50.120069 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:50.120217 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 12:42:50.120277 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:42:50.120440 31689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [term 1 LEADER]: Becoming Leader. State: Replica: 8a821abc91dc4e67b726baa767314a2c, State: Running, Role: LEADER
I20260926 12:42:50.120832 31689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a821abc91dc4e67b726baa767314a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33363 } }
I20260926 12:42:50.121824 31679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:50.123165 31690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a821abc91dc4e67b726baa767314a2c. Latest consensus state: current_term: 1 leader_uuid: "8a821abc91dc4e67b726baa767314a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a821abc91dc4e67b726baa767314a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33363 } } }
I20260926 12:42:50.123303 31690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:50.123659 31691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a821abc91dc4e67b726baa767314a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a821abc91dc4e67b726baa767314a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33363 } } }
I20260926 12:42:50.123788 31691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c [sys.catalog]: This master's current role is: LEADER
W20260926 12:42:50.126520 31704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:42:50.126627 31704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:42:50.126716 31699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:50.127635 31699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:50.133620 31699 catalog_manager.cc:1347] Generated new cluster ID: 28a507a8ca6244c1b4f5347484d7b06b
I20260926 12:42:50.133703 31699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:50.146974 31699 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:50.147313 31699 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:50.154000 31699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a821abc91dc4e67b726baa767314a2c: Generated new TSK 0
I20260926 12:42:50.154909 31699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:50.164463 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 31611
2026-09-26T12:42:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:50 dist-test-slave-m3pj krb5kdc[31710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:50 dist-test-slave-m3pj krb5kdc[31710](info): set up 2 sockets
Sep 26 12:42:50 dist-test-slave-m3pj krb5kdc[31710](info): commencing operation
krb5kdc: starting...
W20260926 12:42:51.212291 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:51 dist-test-slave-m3pj krb5kdc[31710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:51.347813 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:51.466202 31723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:51.466557 31723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:51.466634 31723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:51.470919 31723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:51.471032 31723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:51.471067 31723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:51.471107 31723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:51.471140 31723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:51.476856 31723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:51.478039 31723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:51.479128 31723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:51.488552 31728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:51.488554 31731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:51.488552 31729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:51.488853 31723 server_base.cc:1034] running on GCE node
I20260926 12:42:51.489665 31723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:51.491007 31723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:51.492274 31723 hybrid_clock.cc:648] HybridClock initialized: now 1790426571492248 us; error 52 us; skew 500 ppm
Sep 26 12:42:51 dist-test-slave-m3pj krb5kdc[31710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:51.495347 31723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:51.496454 31723 webserver.cc:460] Webserver started at http://127.0.0.1:42223/ using document root <none> and password file <none>
I20260926 12:42:51.497081 31723 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:51.497179 31723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:51.497468 31723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:51.500367 31723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "077826f593004a29b692691162ff2a41"
format_stamp: "Formatted at 2026-09-26 12:42:51 on dist-test-slave-m3pj"
server_key: "5374da3e906646b7064a6f4ac89b2ebf"
server_key_iv: "8f60f6f442078625dc9496d7d1beb634"
server_key_version: "encryptionkey@0"
I20260926 12:42:51.500949 31723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "077826f593004a29b692691162ff2a41"
format_stamp: "Formatted at 2026-09-26 12:42:51 on dist-test-slave-m3pj"
server_key: "5374da3e906646b7064a6f4ac89b2ebf"
server_key_iv: "8f60f6f442078625dc9496d7d1beb634"
server_key_version: "encryptionkey@0"
I20260926 12:42:51.505167 31723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:51.508016 31738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:51.509454 31723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:51.509706 31723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "077826f593004a29b692691162ff2a41"
format_stamp: "Formatted at 2026-09-26 12:42:51 on dist-test-slave-m3pj"
server_key: "5374da3e906646b7064a6f4ac89b2ebf"
server_key_iv: "8f60f6f442078625dc9496d7d1beb634"
server_key_version: "encryptionkey@0"
I20260926 12:42:51.509848 31723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:51.528008 31723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:51.531839 31723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:51.532085 31723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:51.540016 31723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40465
I20260926 12:42:51.540081 31790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40465 every 8 connection(s)
I20260926 12:42:51.541162 31723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:51.544116 31791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:51.545260 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 31723
I20260926 12:42:51.545447 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:51.545782 11138 external_mini_cluster.cc:1428] Setting key 795ef014ba4c6c9d2c604560e2b10495
I20260926 12:42:51.551220 31791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41: Bootstrap starting.
I20260926 12:42:51.554272 31791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:51.555310 31791 log.cc:824] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:42:51 dist-test-slave-m3pj krb5kdc[31710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:42:51.557534 31791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41: No bootstrap required, opened a new log
I20260926 12:42:51.561285 31791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077826f593004a29b692691162ff2a41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260926 12:42:51.561556 31791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:51.561637 31791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077826f593004a29b692691162ff2a41, State: Initialized, Role: FOLLOWER
I20260926 12:42:51.562103 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:51.547256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49125 (local address 127.0.0.1:40465)
0926 12:42:51.547774 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:51.547796 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:51.547834 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:42:51.548531 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:51.548550 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:51.548922 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:51.549235 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:51.550216 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:51.551453 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:51.552740 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:51.553268 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:51.553462 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:51.556748 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:51.556755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:51.556771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:51.556800 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:51.558794 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:51.559315 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:51.559321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:51.559326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:51.559448 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:51.559803 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:51.559813 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:51.559814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:51.560252 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:51.560483 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:51.560828 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 12:42:51.561157 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":168,"threads_started":1}
I20260926 12:42:51.562143 31791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077826f593004a29b692691162ff2a41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260926 12:42:51.562381 31791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:51.562472 31791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:51.562577 31791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:51.563778 31791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077826f593004a29b692691162ff2a41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260926 12:42:51.564164 31791 leader_election.cc:302] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077826f593004a29b692691162ff2a41; no voters:
I20260926 12:42:51.564546 31791 leader_election.cc:288] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:51.564687 31796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:51.564903 31796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [term 1 LEADER]: Becoming Leader. State: Replica: 077826f593004a29b692691162ff2a41, State: Running, Role: LEADER
I20260926 12:42:51.565227 31796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077826f593004a29b692691162ff2a41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260926 12:42:51.565562 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:42:51.565979 31791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:51.566440 31798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077826f593004a29b692691162ff2a41. Latest consensus state: current_term: 1 leader_uuid: "077826f593004a29b692691162ff2a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077826f593004a29b692691162ff2a41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } } }
I20260926 12:42:51.566558 31798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:51.566915 31797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "077826f593004a29b692691162ff2a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077826f593004a29b692691162ff2a41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } } }
I20260926 12:42:51.566992 31797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:51.567255 31800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:51.570663 31800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:51.577284 31800 catalog_manager.cc:1347] Generated new cluster ID: 1f5c6ac017294ffc9a8e17413195e5b9
I20260926 12:42:51.577358 31800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:51.578516 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:51.568567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46833 (local address 127.0.0.1:40465)
0926 12:42:51.568682 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:51.568685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:51.569086 (+ 401us) server_negotiation.cc:407] Connection header received
0926 12:42:51.569239 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:51.569243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:51.569301 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:51.569382 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:51.570242 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:51.571029 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:51.572413 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:51.572842 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:51.572955 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:51.574795 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:51.574799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:51.574802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:51.574831 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:51.576651 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:51.577167 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:51.577171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:51.577172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:51.577267 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:51.577604 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:51.577608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:51.577609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:51.577807 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:51.577946 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:51.578199 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 12:42:51.578323 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:42:51.591679 31800 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:51.593178 31800 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:51.605155 31800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077826f593004a29b692691162ff2a41: Generated new TSK 0
I20260926 12:42:51.606102 31800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:51.614282 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 31723
2026-09-26T12:42:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:51 dist-test-slave-m3pj krb5kdc[31822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:51 dist-test-slave-m3pj krb5kdc[31822](info): set up 2 sockets
Sep 26 12:42:51 dist-test-slave-m3pj krb5kdc[31822](info): commencing operation
krb5kdc: starting...
W20260926 12:42:52.652498 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:52 dist-test-slave-m3pj krb5kdc[31822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:52.782433 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:52.894737 31835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:52.895175 31835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:52.895221 31835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:52.899437 31835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:52.899534 31835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:52.899556 31835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:52.899574 31835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:52.899590 31835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:52.905155 31835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:52.906273 31835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:52.907290 31835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:42:52.916178 31835 server_base.cc:1034] running on GCE node
W20260926 12:42:52.916167 31841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:52.916167 31840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:52.916471 31843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:52.917203 31835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:52.918365 31835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:52.919615 31835 hybrid_clock.cc:648] HybridClock initialized: now 1790426572919591 us; error 48 us; skew 500 ppm
Sep 26 12:42:52 dist-test-slave-m3pj krb5kdc[31822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:52.922593 31835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:52.923699 31835 webserver.cc:460] Webserver started at http://127.0.0.1:36517/ using document root <none> and password file <none>
I20260926 12:42:52.924310 31835 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:52.924405 31835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:52.924657 31835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:52.926856 31835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "9b97e8cc4fa04ee5856a9410ad883b76"
format_stamp: "Formatted at 2026-09-26 12:42:52 on dist-test-slave-m3pj"
server_key: "457ede87475982220eaaa5487795a546"
server_key_iv: "9d28e2fa644b46a528506dc3863175ed"
server_key_version: "encryptionkey@0"
I20260926 12:42:52.927486 31835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "9b97e8cc4fa04ee5856a9410ad883b76"
format_stamp: "Formatted at 2026-09-26 12:42:52 on dist-test-slave-m3pj"
server_key: "457ede87475982220eaaa5487795a546"
server_key_iv: "9d28e2fa644b46a528506dc3863175ed"
server_key_version: "encryptionkey@0"
I20260926 12:42:52.931959 31835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:52.934877 31850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:52.936698 31835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:52.936856 31835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "9b97e8cc4fa04ee5856a9410ad883b76"
format_stamp: "Formatted at 2026-09-26 12:42:52 on dist-test-slave-m3pj"
server_key: "457ede87475982220eaaa5487795a546"
server_key_iv: "9d28e2fa644b46a528506dc3863175ed"
server_key_version: "encryptionkey@0"
I20260926 12:42:52.937004 31835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:52.953182 31835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:52.956723 31835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:52.956948 31835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:52.964334 31835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39075
I20260926 12:42:52.964360 31902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39075 every 8 connection(s)
I20260926 12:42:52.965394 31835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:52.968422 31903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:52.969671 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 31835
I20260926 12:42:52.969805 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:52.970144 11138 external_mini_cluster.cc:1428] Setting key 6f54f4ad6d73a80824808f625dbf8f6c
I20260926 12:42:52.975167 31903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76: Bootstrap starting.
I20260926 12:42:52.977762 31903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:52.978598 31903 log.cc:824] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:52.980225 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:52.971791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50973 (local address 127.0.0.1:39075)
0926 12:42:52.972387 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:52.972407 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:52.972451 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:42:52.973147 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:52.973165 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:52.973555 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:52.973957 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:52.974468 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:52.975592 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:52.976534 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:52.977042 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:52.977231 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:52.978196 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:52.978202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:52.978218 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:52.978638 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:52.978696 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:52.978720 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:52.978841 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:52.979025 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 12:42:52.979284 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":182,"threads_started":1}
I20260926 12:42:52.981060 31903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76: No bootstrap required, opened a new log
I20260926 12:42:52.983467 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:42:52.985256 31903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b97e8cc4fa04ee5856a9410ad883b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39075 } }
I20260926 12:42:52.985549 31903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:52.985661 31903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b97e8cc4fa04ee5856a9410ad883b76, State: Initialized, Role: FOLLOWER
I20260926 12:42:52.986141 31903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b97e8cc4fa04ee5856a9410ad883b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39075 } }
I20260926 12:42:52.986302 31903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:52.986430 31903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:52.986555 31903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:52.987650 31903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b97e8cc4fa04ee5856a9410ad883b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39075 } }
I20260926 12:42:52.988070 31903 leader_election.cc:302] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b97e8cc4fa04ee5856a9410ad883b76; no voters:
I20260926 12:42:52.988451 31903 leader_election.cc:288] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:52.988566 31913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:52.988780 31913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [term 1 LEADER]: Becoming Leader. State: Replica: 9b97e8cc4fa04ee5856a9410ad883b76, State: Running, Role: LEADER
I20260926 12:42:52.989218 31913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b97e8cc4fa04ee5856a9410ad883b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39075 } }
I20260926 12:42:52.990031 31903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:52.990551 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:52.985682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50137 (local address 127.0.0.1:39075)
0926 12:42:52.986166 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:52.986169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:52.986180 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:42:52.986214 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:52.986216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:52.986254 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:52.986349 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:52.986706 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:52.987311 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:52.988515 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:52.988926 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:52.989029 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:52.990041 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:52.990045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:52.990047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:52.990063 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:52.990093 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:52.990108 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:52.990220 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:52.990322 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 12:42:52.990387 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260926 12:42:52.990658 31913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b97e8cc4fa04ee5856a9410ad883b76. Latest consensus state: current_term: 1 leader_uuid: "9b97e8cc4fa04ee5856a9410ad883b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b97e8cc4fa04ee5856a9410ad883b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39075 } } }
I20260926 12:42:52.990940 31913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:52.991472 31914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b97e8cc4fa04ee5856a9410ad883b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b97e8cc4fa04ee5856a9410ad883b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39075 } } }
I20260926 12:42:52.991991 31914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76 [sys.catalog]: This master's current role is: LEADER
W20260926 12:42:52.994746 31928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:42:52.994853 31928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:42:52.994922 31922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:52.995785 31922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:53.002354 31922 catalog_manager.cc:1347] Generated new cluster ID: cab3f8aef8bc4672b323ee7fa36ed895
I20260926 12:42:53.002449 31922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:53.034849 31922 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:53.035986 31922 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:53.045754 31922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b97e8cc4fa04ee5856a9410ad883b76: Generated new TSK 0
I20260926 12:42:53.046499 31922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:53.056241 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 31835
2026-09-26T12:42:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:53 dist-test-slave-m3pj krb5kdc[31934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:53 dist-test-slave-m3pj krb5kdc[31934](info): set up 2 sockets
Sep 26 12:42:53 dist-test-slave-m3pj krb5kdc[31934](info): commencing operation
krb5kdc: starting...
W20260926 12:42:54.120834 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:54 dist-test-slave-m3pj krb5kdc[31934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:54.254572 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:54.373700 31947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:54.374081 31947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:54.374191 31947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:54.378717 31947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:54.378842 31947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:54.378880 31947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:54.378917 31947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:54.378978 31947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:54.385102 31947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:54.386396 31947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:54.387593 31947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:54.397706 31953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:54.397716 31955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:54.397783 31947 server_base.cc:1034] running on GCE node
W20260926 12:42:54.397706 31952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:54.398587 31947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:54.400080 31947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:54.401280 31947 hybrid_clock.cc:648] HybridClock initialized: now 1790426574401267 us; error 32 us; skew 500 ppm
Sep 26 12:42:54 dist-test-slave-m3pj krb5kdc[31934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:54.404433 31947 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:54.405474 31947 webserver.cc:460] Webserver started at http://127.0.0.1:41343/ using document root <none> and password file <none>
I20260926 12:42:54.406095 31947 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:54.406163 31947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:54.406392 31947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:54.408751 31947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a"
format_stamp: "Formatted at 2026-09-26 12:42:54 on dist-test-slave-m3pj"
server_key: "07c4bdee5b684fd3bc02eab00944fbcb"
server_key_iv: "1b14eadbc32d365dad06489895cd66cc"
server_key_version: "encryptionkey@0"
I20260926 12:42:54.409382 31947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a"
format_stamp: "Formatted at 2026-09-26 12:42:54 on dist-test-slave-m3pj"
server_key: "07c4bdee5b684fd3bc02eab00944fbcb"
server_key_iv: "1b14eadbc32d365dad06489895cd66cc"
server_key_version: "encryptionkey@0"
I20260926 12:42:54.413872 31947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:42:54.416682 31962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:54.417986 31947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:54.418148 31947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a"
format_stamp: "Formatted at 2026-09-26 12:42:54 on dist-test-slave-m3pj"
server_key: "07c4bdee5b684fd3bc02eab00944fbcb"
server_key_iv: "1b14eadbc32d365dad06489895cd66cc"
server_key_version: "encryptionkey@0"
I20260926 12:42:54.418283 31947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:54.423269 31947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:54.426249 31947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:54.426452 31947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:54.433784 31947 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37535
I20260926 12:42:54.433838 32014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37535 every 8 connection(s)
I20260926 12:42:54.434818 31947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:54.437794 32015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:54.441780 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 31947
I20260926 12:42:54.441963 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:54.442272 11138 external_mini_cluster.cc:1428] Setting key 2dee97c4714265f99628c09a236ed1e1
I20260926 12:42:54.444146 32015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a: Bootstrap starting.
I20260926 12:42:54.446705 32015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:54.447723 32015 log.cc:824] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:54.447922 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:54.443779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56801 (local address 127.0.0.1:37535)
0926 12:42:54.444301 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:54.444318 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:54.444355 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:42:54.445028 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:54.445045 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:54.445349 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:54.445737 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:54.446557 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:54.446562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:54.446574 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:54.446585 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:54.446639 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:54.446659 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:54.446778 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:54.446884 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 12:42:54.447066 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":151,"threads_started":1}
I20260926 12:42:54.450186 32015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a: No bootstrap required, opened a new log
I20260926 12:42:54.451119 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:42:54.454942 32015 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260926 12:42:54.455233 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:54.453404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35645 (local address 127.0.0.1:37535)
0926 12:42:54.453555 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:54.453558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:54.453715 (+ 157us) server_negotiation.cc:407] Connection header received
0926 12:42:54.453845 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:54.453847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:54.453880 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:54.453948 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:54.454725 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:54.454729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:54.454731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:54.454742 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:54.454769 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:54.454779 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:54.454892 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:54.455008 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 12:42:54.455045 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 12:42:54.455328 32015 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:54.455416 32015 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f4bf97cb1d4fa8ba8de17ecb71893a, State: Initialized, Role: FOLLOWER
I20260926 12:42:54.455960 32015 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260926 12:42:54.456133 32015 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:54.456249 32015 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:54.456372 32015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:54.457403 32015 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260926 12:42:54.457793 32015 leader_election.cc:302] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8f4bf97cb1d4fa8ba8de17ecb71893a; no voters:
I20260926 12:42:54.458204 32015 leader_election.cc:288] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:54.458302 32025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:54.458607 32025 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [term 1 LEADER]: Becoming Leader. State: Replica: c8f4bf97cb1d4fa8ba8de17ecb71893a, State: Running, Role: LEADER
I20260926 12:42:54.458976 32025 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260926 12:42:54.459633 32015 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:54.460843 32027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f4bf97cb1d4fa8ba8de17ecb71893a. Latest consensus state: current_term: 1 leader_uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } } }
I20260926 12:42:54.460974 32027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:54.461457 32034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:54.461632 32026 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f4bf97cb1d4fa8ba8de17ecb71893a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } } }
I20260926 12:42:54.462363 32026 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:54.465240 32034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:54.472000 32034 catalog_manager.cc:1347] Generated new cluster ID: b881f6052e1a40b484453752b9e7e7ec
I20260926 12:42:54.472101 32034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:54.504067 32034 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:54.504420 32034 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:54.512768 32034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f4bf97cb1d4fa8ba8de17ecb71893a: Generated new TSK 0
I20260926 12:42:54.513523 32034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:54.524326 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 31947
2026-09-26T12:42:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:54 dist-test-slave-m3pj krb5kdc[32046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:54 dist-test-slave-m3pj krb5kdc[32046](info): set up 2 sockets
Sep 26 12:42:54 dist-test-slave-m3pj krb5kdc[32046](info): commencing operation
krb5kdc: starting...
W20260926 12:42:55.564160 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:55 dist-test-slave-m3pj krb5kdc[32046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:55.692891 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:55.809137 32059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:55.809489 32059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:55.809571 32059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:55.813905 32059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:55.814051 32059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:55.814090 32059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:55.814129 32059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:55.814163 32059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:55.820099 32059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:55.821359 32059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:55.822482 32059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:55.831532 32065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:55.831552 32064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:55.831601 32059 server_base.cc:1034] running on GCE node
W20260926 12:42:55.831561 32067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:55.832329 32059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:55.833580 32059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:55.834803 32059 hybrid_clock.cc:648] HybridClock initialized: now 1790426575834767 us; error 46 us; skew 500 ppm
Sep 26 12:42:55 dist-test-slave-m3pj krb5kdc[32046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:55.843904 32059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:55.845100 32059 webserver.cc:460] Webserver started at http://127.0.0.1:34125/ using document root <none> and password file <none>
I20260926 12:42:55.845726 32059 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:55.845824 32059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:55.846081 32059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:55.848543 32059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "01ff2bf1116a4500b390c5daff9d9538"
format_stamp: "Formatted at 2026-09-26 12:42:55 on dist-test-slave-m3pj"
server_key: "7163ffdc7e397410c53107d753891669"
server_key_iv: "613b0b21c41beffda52dadb23c4f5136"
server_key_version: "encryptionkey@0"
I20260926 12:42:55.849179 32059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "01ff2bf1116a4500b390c5daff9d9538"
format_stamp: "Formatted at 2026-09-26 12:42:55 on dist-test-slave-m3pj"
server_key: "7163ffdc7e397410c53107d753891669"
server_key_iv: "613b0b21c41beffda52dadb23c4f5136"
server_key_version: "encryptionkey@0"
I20260926 12:42:55.853668 32059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:42:55.856637 32074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:55.858112 32059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:55.858294 32059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "01ff2bf1116a4500b390c5daff9d9538"
format_stamp: "Formatted at 2026-09-26 12:42:55 on dist-test-slave-m3pj"
server_key: "7163ffdc7e397410c53107d753891669"
server_key_iv: "613b0b21c41beffda52dadb23c4f5136"
server_key_version: "encryptionkey@0"
I20260926 12:42:55.858436 32059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:55.871655 32059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:55.875639 32059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:55.875878 32059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:55.883590 32059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34691
I20260926 12:42:55.883616 32126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34691 every 8 connection(s)
I20260926 12:42:55.884634 32059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:55.887646 32127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:55.890249 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 32059
I20260926 12:42:55.890389 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:55.890717 11138 external_mini_cluster.cc:1428] Setting key 5b49d5f654135e3aef1b2dfd79a33c43
I20260926 12:42:55.894361 32127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538: Bootstrap starting.
I20260926 12:42:55.896970 32127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:55.897790 32127 log.cc:824] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:55.899967 32127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538: No bootstrap required, opened a new log
Sep 26 12:42:55 dist-test-slave-m3pj krb5kdc[32046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:42:55.903421 32127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ff2bf1116a4500b390c5daff9d9538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260926 12:42:55.903721 32127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:55.903805 32127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ff2bf1116a4500b390c5daff9d9538, State: Initialized, Role: FOLLOWER
I20260926 12:42:55.904313 32127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ff2bf1116a4500b390c5daff9d9538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260926 12:42:55.904469 32127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:55.904534 32127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:55.904634 32127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:55.905576 32127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ff2bf1116a4500b390c5daff9d9538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260926 12:42:55.905928 32127 leader_election.cc:302] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01ff2bf1116a4500b390c5daff9d9538; no voters:
I20260926 12:42:55.906281 32127 leader_election.cc:288] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:55.906356 32132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:55.906625 32132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [term 1 LEADER]: Becoming Leader. State: Replica: 01ff2bf1116a4500b390c5daff9d9538, State: Running, Role: LEADER
I20260926 12:42:55.907008 32132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ff2bf1116a4500b390c5daff9d9538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } }
I20260926 12:42:55.907647 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:55.892446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:47575 (local address 127.0.0.1:34691)
0926 12:42:55.892965 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:55.892981 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:55.893021 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:42:55.893741 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:55.893761 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:55.894154 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:55.894538 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:55.895742 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:55.896780 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:55.897844 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:55.898403 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:55.898607 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:55.901555 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:55.901560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:55.901576 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:55.901609 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:55.904222 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:55.904876 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:55.904881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:55.904886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:55.904997 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:55.905356 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:55.905364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:55.905365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:55.905809 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:55.906046 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:55.906364 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 12:42:55.906617 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":143,"threads_started":1}
I20260926 12:42:55.907753 32127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:55.908291 32132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01ff2bf1116a4500b390c5daff9d9538. Latest consensus state: current_term: 1 leader_uuid: "01ff2bf1116a4500b390c5daff9d9538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ff2bf1116a4500b390c5daff9d9538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } } }
I20260926 12:42:55.908465 32132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:55.909613 32132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01ff2bf1116a4500b390c5daff9d9538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01ff2bf1116a4500b390c5daff9d9538" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34691 } } }
I20260926 12:42:55.909761 32132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:55.910939 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 12:42:55.913364 32147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:42:55.913497 32147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:42:55.913633 32138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:55.914512 32138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:55.921370 32138 catalog_manager.cc:1347] Generated new cluster ID: 5869f3e4f0394240b42819066949a624
I20260926 12:42:55.921439 32138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:55.923902 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:55.913347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:50341 (local address 127.0.0.1:34691)
0926 12:42:55.913464 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:55.913470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:55.913484 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:42:55.913524 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:55.913526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:55.913568 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:55.913661 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:55.914324 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:55.914840 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:55.915721 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:55.916002 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:55.916116 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:55.917745 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:55.917749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:55.917752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:42:55.917778 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:55.919606 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:55.920109 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:55.920113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:55.920114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:55.920195 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:42:55.920528 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:42:55.920531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:42:55.920533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:42:55.920714 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:55.920868 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:55.921189 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 12:42:55.921310 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:42:55.940361 32138 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:55.941414 32138 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:55.948403 32138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01ff2bf1116a4500b390c5daff9d9538: Generated new TSK 0
I20260926 12:42:55.949199 32138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:55.958284 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 32059
W20260926 12:42:55.963227 32044 connection.cc:535] client connection to 127.0.0.1:34691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34691): Connection reset by peer (error 104)
2026-09-26T12:42:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:56 dist-test-slave-m3pj krb5kdc[32158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:56 dist-test-slave-m3pj krb5kdc[32158](info): set up 2 sockets
Sep 26 12:42:56 dist-test-slave-m3pj krb5kdc[32158](info): commencing operation
krb5kdc: starting...
W20260926 12:42:57.024399 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:57 dist-test-slave-m3pj krb5kdc[32158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:57.159696 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:57.275249 32171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:57.275604 32171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:57.275664 32171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:57.280226 32171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:57.280301 32171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:57.280323 32171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:57.280346 32171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:57.280362 32171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:57.285837 32171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:57.286952 32171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:57.288077 32171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:57.297794 32177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:57.298043 32179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:57.298180 32171 server_base.cc:1034] running on GCE node
W20260926 12:42:57.297870 32176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:57.298949 32171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:57.300272 32171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:57.301489 32171 hybrid_clock.cc:648] HybridClock initialized: now 1790426577301473 us; error 28 us; skew 500 ppm
Sep 26 12:42:57 dist-test-slave-m3pj krb5kdc[32158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:57.304457 32171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:57.305418 32171 webserver.cc:460] Webserver started at http://127.0.0.1:44101/ using document root <none> and password file <none>
I20260926 12:42:57.306034 32171 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:57.306123 32171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:57.306350 32171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:57.308553 32171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "041a96dbba5b4356a18facf9abe48913"
format_stamp: "Formatted at 2026-09-26 12:42:57 on dist-test-slave-m3pj"
server_key: "c4a141d36a30937752c229b6d69fef36"
server_key_iv: "44206c6dcf403781874daefc9b01ca57"
server_key_version: "encryptionkey@0"
I20260926 12:42:57.309118 32171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "041a96dbba5b4356a18facf9abe48913"
format_stamp: "Formatted at 2026-09-26 12:42:57 on dist-test-slave-m3pj"
server_key: "c4a141d36a30937752c229b6d69fef36"
server_key_iv: "44206c6dcf403781874daefc9b01ca57"
server_key_version: "encryptionkey@0"
I20260926 12:42:57.313053 32171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:42:57.315857 32186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:57.317279 32171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:57.317473 32171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "041a96dbba5b4356a18facf9abe48913"
format_stamp: "Formatted at 2026-09-26 12:42:57 on dist-test-slave-m3pj"
server_key: "c4a141d36a30937752c229b6d69fef36"
server_key_iv: "44206c6dcf403781874daefc9b01ca57"
server_key_version: "encryptionkey@0"
I20260926 12:42:57.317619 32171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:57.338792 32171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:57.343070 32171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:57.343323 32171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:57.353516 32171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40773
I20260926 12:42:57.353598 32238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40773 every 8 connection(s)
I20260926 12:42:57.354646 32171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:57.356838 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 32171
I20260926 12:42:57.356968 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:57.357334 11138 external_mini_cluster.cc:1428] Setting key ee8b6bf9401ab95d78e8039cfcb5c51c
I20260926 12:42:57.358285 32239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:57.366062 32239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913: Bootstrap starting.
I20260926 12:42:57.368507 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:57.358892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:42815 (local address 127.0.0.1:40773)
0926 12:42:57.359370 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:57.359390 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:57.359432 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:42:57.360353 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:57.360379 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:57.360861 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:57.361292 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:57.361741 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:57.363301 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:57.364359 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:57.365117 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:57.365279 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:57.366293 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:57.366300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:57.366317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:57.366846 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:57.366921 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:57.366953 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:57.367115 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:57.367320 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:42:57.367482 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":145,"threads_started":1}
I20260926 12:42:57.369036 32239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:57.369971 32239 log.cc:824] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:57.372112 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:42:57.372784 32239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913: No bootstrap required, opened a new log
I20260926 12:42:57.377902 32239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a96dbba5b4356a18facf9abe48913" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260926 12:42:57.378312 32239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:57.378388 32239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041a96dbba5b4356a18facf9abe48913, State: Initialized, Role: FOLLOWER
I20260926 12:42:57.378940 32239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a96dbba5b4356a18facf9abe48913" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260926 12:42:57.379134 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:57.374518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:35411 (local address 127.0.0.1:40773)
0926 12:42:57.374674 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:57.374679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:57.374820 (+ 141us) server_negotiation.cc:407] Connection header received
0926 12:42:57.374932 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:57.374935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:57.374988 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:57.375088 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:57.375349 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:57.376254 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:57.377184 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:42:57.377503 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:42:57.377614 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:42:57.378604 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:57.378608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:57.378610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:57.378625 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:57.378653 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:57.378663 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:57.378766 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:57.378926 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 12:42:57.378973 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:42:57.379142 32239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:57.379271 32239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:57.379386 32239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:57.380633 32239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a96dbba5b4356a18facf9abe48913" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260926 12:42:57.381031 32239 leader_election.cc:302] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 041a96dbba5b4356a18facf9abe48913; no voters:
I20260926 12:42:57.381408 32239 leader_election.cc:288] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:57.381598 32249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:57.381870 32249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [term 1 LEADER]: Becoming Leader. State: Replica: 041a96dbba5b4356a18facf9abe48913, State: Running, Role: LEADER
I20260926 12:42:57.382202 32249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a96dbba5b4356a18facf9abe48913" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } }
I20260926 12:42:57.382858 32239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:57.384043 32251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041a96dbba5b4356a18facf9abe48913. Latest consensus state: current_term: 1 leader_uuid: "041a96dbba5b4356a18facf9abe48913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a96dbba5b4356a18facf9abe48913" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } } }
I20260926 12:42:57.384209 32251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:57.384835 32259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:57.384794 32250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "041a96dbba5b4356a18facf9abe48913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041a96dbba5b4356a18facf9abe48913" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40773 } } }
I20260926 12:42:57.384938 32250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:57.388629 32259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:57.394846 32259 catalog_manager.cc:1347] Generated new cluster ID: bbf26a88fe7648dfa5e9f0f32a8e3ad5
I20260926 12:42:57.394951 32259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:57.409991 32259 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:57.411065 32259 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:57.417701 32259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 041a96dbba5b4356a18facf9abe48913: Generated new TSK 0
I20260926 12:42:57.418489 32259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:57.425468 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 32171
W20260926 12:42:57.430989 32156 connection.cc:535] client connection to 127.0.0.1:40773 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40773): Connection reset by peer (error 104)
2026-09-26T12:42:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:57 dist-test-slave-m3pj krb5kdc[32270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:57 dist-test-slave-m3pj krb5kdc[32270](info): set up 2 sockets
Sep 26 12:42:57 dist-test-slave-m3pj krb5kdc[32270](info): commencing operation
krb5kdc: starting...
W20260926 12:42:58.476248 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:42:58 dist-test-slave-m3pj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:42:58.604914 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:42:58.721722 32283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:42:58.722056 32283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:42:58.722141 32283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:42:58.726209 32283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:42:58.726318 32283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:42:58.726357 32283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:42:58.726418 32283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:42:58.726464 32283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:42:58.731914 32283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:42:58.733096 32283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:42:58.734223 32283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:42:58.743295 32288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:58.743383 32283 server_base.cc:1034] running on GCE node
W20260926 12:42:58.743287 32289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:42:58.743326 32291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:42:58.744132 32283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:42:58.745283 32283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:42:58.746501 32283 hybrid_clock.cc:648] HybridClock initialized: now 1790426578746442 us; error 30 us; skew 500 ppm
Sep 26 12:42:58 dist-test-slave-m3pj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:42:58.749432 32283 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:42:58.750362 32283 webserver.cc:460] Webserver started at http://127.0.0.1:44521/ using document root <none> and password file <none>
I20260926 12:42:58.750892 32283 fs_manager.cc:359] Metadata directory not provided
I20260926 12:42:58.750993 32283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:42:58.751233 32283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:42:58.753439 32283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "d45376d919a34a4c915048c8e22775a6"
format_stamp: "Formatted at 2026-09-26 12:42:58 on dist-test-slave-m3pj"
server_key: "54fba83b3282d7b88a990c3c15ed6741"
server_key_iv: "752c3984c3870407feec057649c92186"
server_key_version: "encryptionkey@0"
I20260926 12:42:58.754029 32283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "d45376d919a34a4c915048c8e22775a6"
format_stamp: "Formatted at 2026-09-26 12:42:58 on dist-test-slave-m3pj"
server_key: "54fba83b3282d7b88a990c3c15ed6741"
server_key_iv: "752c3984c3870407feec057649c92186"
server_key_version: "encryptionkey@0"
I20260926 12:42:58.758042 32283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:42:58.760715 32298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:42:58.762080 32283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:42:58.762235 32283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "d45376d919a34a4c915048c8e22775a6"
format_stamp: "Formatted at 2026-09-26 12:42:58 on dist-test-slave-m3pj"
server_key: "54fba83b3282d7b88a990c3c15ed6741"
server_key_iv: "752c3984c3870407feec057649c92186"
server_key_version: "encryptionkey@0"
I20260926 12:42:58.762372 32283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:42:58.767216 32283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:42:58.770411 32283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:42:58.770622 32283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:42:58.778355 32283 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43921
I20260926 12:42:58.778376 32350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43921 every 8 connection(s)
I20260926 12:42:58.779356 32283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:42:58.781699 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 32283
I20260926 12:42:58.781870 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:42:58.782231 11138 external_mini_cluster.cc:1428] Setting key 7ed1821118a8fd92a0b326163fc74d6b
I20260926 12:42:58.783232 32351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:42:58.787668 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:58.783730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:47709 (local address 127.0.0.1:43921)
0926 12:42:58.784189 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:58.784205 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:58.784240 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:42:58.784873 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:58.784890 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:58.785255 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:58.785612 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:58.786397 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:58.786402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:58.786424 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:58.786435 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:58.786488 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:58.786508 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:58.786661 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:58.786728 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 12:42:58.786895 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260926 12:42:58.789873 32351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6: Bootstrap starting.
I20260926 12:42:58.790525 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:42:58.792300 32351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6: Neither blocks nor log segments found. Creating new log.
I20260926 12:42:58.793296 32351 log.cc:824] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:42:58.794492 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:42:58.792630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:38997 (local address 127.0.0.1:43921)
0926 12:42:58.792769 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:42:58.792773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:42:58.792995 (+ 222us) server_negotiation.cc:407] Connection header received
0926 12:42:58.793112 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:42:58.793115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:42:58.793147 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:42:58.793210 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:42:58.793986 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:42:58.793993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:42:58.793999 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:42:58.794013 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:42:58.794043 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:42:58.794059 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:42:58.794160 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:42:58.794314 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 12:42:58.794349 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:42:58.795719 32351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6: No bootstrap required, opened a new log
I20260926 12:42:58.799185 32351 raft_consensus.cc:348] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45376d919a34a4c915048c8e22775a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43921 } }
I20260926 12:42:58.799449 32351 raft_consensus.cc:374] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:42:58.799533 32351 raft_consensus.cc:729] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45376d919a34a4c915048c8e22775a6, State: Initialized, Role: FOLLOWER
I20260926 12:42:58.799997 32351 consensus_queue.cc:260] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45376d919a34a4c915048c8e22775a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43921 } }
I20260926 12:42:58.800150 32351 raft_consensus.cc:388] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:42:58.800218 32351 raft_consensus.cc:482] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:42:58.800313 32351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:42:58.801266 32351 raft_consensus.cc:504] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45376d919a34a4c915048c8e22775a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43921 } }
I20260926 12:42:58.801622 32351 leader_election.cc:302] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d45376d919a34a4c915048c8e22775a6; no voters:
I20260926 12:42:58.801949 32351 leader_election.cc:288] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:42:58.802075 32361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:42:58.802326 32361 raft_consensus.cc:686] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [term 1 LEADER]: Becoming Leader. State: Replica: d45376d919a34a4c915048c8e22775a6, State: Running, Role: LEADER
I20260926 12:42:58.802683 32361 consensus_queue.cc:237] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45376d919a34a4c915048c8e22775a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43921 } }
I20260926 12:42:58.803228 32351 sys_catalog.cc:564] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:42:58.804332 32363 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45376d919a34a4c915048c8e22775a6. Latest consensus state: current_term: 1 leader_uuid: "d45376d919a34a4c915048c8e22775a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45376d919a34a4c915048c8e22775a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43921 } } }
I20260926 12:42:58.804514 32363 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:58.804821 32362 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d45376d919a34a4c915048c8e22775a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45376d919a34a4c915048c8e22775a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43921 } } }
I20260926 12:42:58.804914 32370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:42:58.804975 32362 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:42:58.808403 32370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:42:58.814761 32370 catalog_manager.cc:1347] Generated new cluster ID: 157ad22106394371b182cd6efd4a0a28
I20260926 12:42:58.814862 32370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:42:58.860738 32370 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:42:58.861159 32370 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:42:58.867379 32370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d45376d919a34a4c915048c8e22775a6: Generated new TSK 0
I20260926 12:42:58.868254 32370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:42:58.878347 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 32283
2026-09-26T12:42:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:42:58 dist-test-slave-m3pj krb5kdc[32382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:42:58 dist-test-slave-m3pj krb5kdc[32382](info): set up 2 sockets
Sep 26 12:42:58 dist-test-slave-m3pj krb5kdc[32382](info): commencing operation
krb5kdc: starting...
W20260926 12:42:59.924260 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:00 dist-test-slave-m3pj krb5kdc[32382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:43:00.052700 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:00.167048 32395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:00.167395 32395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:00.167474 32395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:00.172248 32395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:00.172346 32395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:00.172384 32395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:00.172435 32395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:00.172473 32395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:00.178102 32395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:00.179267 32395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:00.180397 32395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:00.188922 32403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:00.189036 32395 server_base.cc:1034] running on GCE node
W20260926 12:43:00.188951 32401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:00.189242 32400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:00.189842 32395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:00.190904 32395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:00.192070 32395 hybrid_clock.cc:648] HybridClock initialized: now 1790426580192050 us; error 31 us; skew 500 ppm
Sep 26 12:43:00 dist-test-slave-m3pj krb5kdc[32382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:00.194715 32395 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:43:00.195662 32395 webserver.cc:460] Webserver started at http://127.0.0.1:44949/ using document root <none> and password file <none>
I20260926 12:43:00.196210 32395 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:00.196297 32395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:00.196530 32395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:00.198513 32395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "c5578637cf6241ad8b0b78bdcb6ee282"
format_stamp: "Formatted at 2026-09-26 12:43:00 on dist-test-slave-m3pj"
server_key: "b82134b5bc31ab4062fd48727399b5f8"
server_key_iv: "4994f38f2a7c361f5d44ddd272509f21"
server_key_version: "encryptionkey@0"
I20260926 12:43:00.199067 32395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "c5578637cf6241ad8b0b78bdcb6ee282"
format_stamp: "Formatted at 2026-09-26 12:43:00 on dist-test-slave-m3pj"
server_key: "b82134b5bc31ab4062fd48727399b5f8"
server_key_iv: "4994f38f2a7c361f5d44ddd272509f21"
server_key_version: "encryptionkey@0"
I20260926 12:43:00.202687 32395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 12:43:00.205055 32410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:00.206336 32395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:43:00.206487 32395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "c5578637cf6241ad8b0b78bdcb6ee282"
format_stamp: "Formatted at 2026-09-26 12:43:00 on dist-test-slave-m3pj"
server_key: "b82134b5bc31ab4062fd48727399b5f8"
server_key_iv: "4994f38f2a7c361f5d44ddd272509f21"
server_key_version: "encryptionkey@0"
I20260926 12:43:00.206600 32395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:00.219252 32395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:00.222327 32395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:00.222548 32395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:00.230298 32395 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45299
I20260926 12:43:00.230321 32462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45299 every 8 connection(s)
I20260926 12:43:00.231309 32395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:00.234275 32463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:00.239585 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 32395
I20260926 12:43:00.239737 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:00.240018 11138 external_mini_cluster.cc:1428] Setting key 920b1e9f961b816a48d7625859b39fd2
I20260926 12:43:00.240471 32463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282: Bootstrap starting.
I20260926 12:43:00.243018 32463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:00.244108 32463 log.cc:824] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:00.246379 32463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282: No bootstrap required, opened a new log
Sep 26 12:43:00 dist-test-slave-m3pj krb5kdc[32382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:43:00.251184 32463 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5578637cf6241ad8b0b78bdcb6ee282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45299 } }
I20260926 12:43:00.251456 32463 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:00.251582 32463 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5578637cf6241ad8b0b78bdcb6ee282, State: Initialized, Role: FOLLOWER
I20260926 12:43:00.252079 32463 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5578637cf6241ad8b0b78bdcb6ee282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45299 } }
I20260926 12:43:00.252243 32463 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:00.252338 32463 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:00.252467 32463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:00.253438 32463 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5578637cf6241ad8b0b78bdcb6ee282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45299 } }
I20260926 12:43:00.253813 32463 leader_election.cc:302] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5578637cf6241ad8b0b78bdcb6ee282; no voters:
I20260926 12:43:00.254207 32463 leader_election.cc:288] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:00.254280 32468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:00.254509 32468 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [term 1 LEADER]: Becoming Leader. State: Replica: c5578637cf6241ad8b0b78bdcb6ee282, State: Running, Role: LEADER
I20260926 12:43:00.255087 32468 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5578637cf6241ad8b0b78bdcb6ee282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45299 } }
I20260926 12:43:00.255725 32463 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:00.256007 32466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:00.241506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:39011 (local address 127.0.0.1:45299)
0926 12:43:00.241922 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:00.241942 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:00.241982 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:43:00.242615 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:00.242632 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:00.242980 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:00.243309 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:00.244348 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:00.245321 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:00.246557 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:00.247030 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:00.247160 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:00.250273 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:00.250278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:00.250291 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:00.250318 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:00.252443 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:00.252961 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:00.252967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:00.252973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:00.253093 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:00.253431 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:00.253435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:00.253436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:00.253887 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:00.254116 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:00.254818 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 12:43:00.255040 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":147,"threads_started":1}
I20260926 12:43:00.256855 32470 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5578637cf6241ad8b0b78bdcb6ee282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5578637cf6241ad8b0b78bdcb6ee282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45299 } } }
I20260926 12:43:00.257043 32470 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:00.257475 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5578637cf6241ad8b0b78bdcb6ee282. Latest consensus state: current_term: 1 leader_uuid: "c5578637cf6241ad8b0b78bdcb6ee282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5578637cf6241ad8b0b78bdcb6ee282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45299 } } }
I20260926 12:43:00.257619 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:00.257915 32477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:00.259892 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:00.261718 32477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:00.267567 32477 catalog_manager.cc:1347] Generated new cluster ID: 722590c07bde4cc8b498de6549063c82
I20260926 12:43:00.267635 32477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:00.272823 32466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:00.263905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:59861 (local address 127.0.0.1:45299)
0926 12:43:00.264009 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:00.264014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:00.264272 (+ 258us) server_negotiation.cc:407] Connection header received
0926 12:43:00.264396 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:00.264400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:00.264460 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:00.264565 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:00.265418 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:00.266394 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:00.267497 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:00.267830 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:00.267949 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:00.269406 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:00.269410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:00.269413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:00.269445 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:00.270943 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:00.271431 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:00.271435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:00.271437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:00.271569 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:00.271918 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:00.271921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:00.271923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:00.272117 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:00.272270 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:00.272548 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 12:43:00.272643 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260926 12:43:00.281358 32477 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:00.282697 32477 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:00.303608 32477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5578637cf6241ad8b0b78bdcb6ee282: Generated new TSK 0
I20260926 12:43:00.304607 32477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:00.312614 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 32395
W20260926 12:43:00.318406 32380 connection.cc:535] client connection to 127.0.0.1:45299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45299): Connection reset by peer (error 104)
2026-09-26T12:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:00 dist-test-slave-m3pj krb5kdc[32494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:00 dist-test-slave-m3pj krb5kdc[32494](info): set up 2 sockets
Sep 26 12:43:00 dist-test-slave-m3pj krb5kdc[32494](info): commencing operation
krb5kdc: starting...
W20260926 12:43:01.372921 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:01 dist-test-slave-m3pj krb5kdc[32494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:43:01.506542 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:01.621281 32507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:01.621598 32507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:01.621642 32507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:01.625988 32507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:01.626071 32507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:01.626091 32507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:01.626109 32507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:01.626125 32507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:01.631901 32507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:01.633064 32507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:01.634161 32507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:43:01.643170 32507 server_base.cc:1034] running on GCE node
W20260926 12:43:01.643114 32515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:01.643198 32512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:01.643142 32513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:01.644037 32507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:01.645280 32507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:01.646507 32507 hybrid_clock.cc:648] HybridClock initialized: now 1790426581646489 us; error 31 us; skew 500 ppm
Sep 26 12:43:01 dist-test-slave-m3pj krb5kdc[32494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:01.655860 32507 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:43:01.656988 32507 webserver.cc:460] Webserver started at http://127.0.0.1:35075/ using document root <none> and password file <none>
I20260926 12:43:01.657660 32507 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:01.657764 32507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:01.658016 32507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:01.660286 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "9dfcc5541a4a4b55867ee5bc7257fc04"
format_stamp: "Formatted at 2026-09-26 12:43:01 on dist-test-slave-m3pj"
server_key: "12456df8c5c7eda36eb7a2b8050626a1"
server_key_iv: "5350859e4cac7f06a1ca202b462bf2b6"
server_key_version: "encryptionkey@0"
I20260926 12:43:01.660859 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "9dfcc5541a4a4b55867ee5bc7257fc04"
format_stamp: "Formatted at 2026-09-26 12:43:01 on dist-test-slave-m3pj"
server_key: "12456df8c5c7eda36eb7a2b8050626a1"
server_key_iv: "5350859e4cac7f06a1ca202b462bf2b6"
server_key_version: "encryptionkey@0"
I20260926 12:43:01.665199 32507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:43:01.668126 32522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:01.669579 32507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:43:01.669749 32507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "9dfcc5541a4a4b55867ee5bc7257fc04"
format_stamp: "Formatted at 2026-09-26 12:43:01 on dist-test-slave-m3pj"
server_key: "12456df8c5c7eda36eb7a2b8050626a1"
server_key_iv: "5350859e4cac7f06a1ca202b462bf2b6"
server_key_version: "encryptionkey@0"
I20260926 12:43:01.669883 32507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:01.684578 32507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:01.688023 32507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:01.688251 32507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:01.695520 32507 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39017
I20260926 12:43:01.695575 32574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39017 every 8 connection(s)
I20260926 12:43:01.696575 32507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:01.699618 32575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:01.703740 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 32507
I20260926 12:43:01.703908 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:01.704224 11138 external_mini_cluster.cc:1428] Setting key 386f47d2efedc789449d88922f2c0c8b
I20260926 12:43:01.705971 32575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04: Bootstrap starting.
I20260926 12:43:01.708246 32575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:01.709000 32575 log.cc:824] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:01.710834 32575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04: No bootstrap required, opened a new log
I20260926 12:43:01.713542 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:01.705699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:59985 (local address 127.0.0.1:39017)
0926 12:43:01.706198 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:01.706213 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:01.706249 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:43:01.706924 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:01.706941 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:01.707283 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:01.707655 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:01.708073 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:01.709192 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:01.710077 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:01.710550 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:01.710723 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:01.711816 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:01.711821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:01.711834 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:01.712233 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:01.712286 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:01.712309 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:01.712459 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:01.712655 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 12:43:01.712800 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":155,"threads_started":1}
I20260926 12:43:01.714893 32575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcc5541a4a4b55867ee5bc7257fc04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } }
I20260926 12:43:01.715168 32575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:01.715216 32575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfcc5541a4a4b55867ee5bc7257fc04, State: Initialized, Role: FOLLOWER
I20260926 12:43:01.715713 32575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcc5541a4a4b55867ee5bc7257fc04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } }
I20260926 12:43:01.715876 32575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:01.715965 32575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:01.716073 32575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:01.716678 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:01.717178 32575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcc5541a4a4b55867ee5bc7257fc04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } }
I20260926 12:43:01.717519 32575 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfcc5541a4a4b55867ee5bc7257fc04; no voters:
I20260926 12:43:01.717840 32575 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:01.717959 32581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:01.718231 32581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [term 1 LEADER]: Becoming Leader. State: Replica: 9dfcc5541a4a4b55867ee5bc7257fc04, State: Running, Role: LEADER
I20260926 12:43:01.718590 32581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcc5541a4a4b55867ee5bc7257fc04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } }
I20260926 12:43:01.719162 32575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:01.719754 32586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfcc5541a4a4b55867ee5bc7257fc04. Latest consensus state: current_term: 1 leader_uuid: "9dfcc5541a4a4b55867ee5bc7257fc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcc5541a4a4b55867ee5bc7257fc04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } } }
I20260926 12:43:01.719894 32586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:01.720450 32588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:01.720944 32585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dfcc5541a4a4b55867ee5bc7257fc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcc5541a4a4b55867ee5bc7257fc04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39017 } } }
I20260926 12:43:01.721047 32585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:01.725227 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:01.720272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:60049 (local address 127.0.0.1:39017)
0926 12:43:01.720371 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:01.720374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:01.720683 (+ 309us) server_negotiation.cc:407] Connection header received
0926 12:43:01.720859 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:01.720862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:01.720910 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:01.720960 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:01.721367 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:01.722138 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:01.723267 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:01.723628 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:01.723730 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:01.724701 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:01.724707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:01.724711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:01.724727 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:01.724762 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:01.724792 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:01.724906 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:01.725036 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 12:43:01.725070 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 12:43:01.727752 32588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:01.733322 32588 catalog_manager.cc:1347] Generated new cluster ID: c47d589ef925497ab27f54c432729b43
I20260926 12:43:01.733435 32588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:01.758738 32588 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:01.759835 32588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:01.766273 32588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dfcc5541a4a4b55867ee5bc7257fc04: Generated new TSK 0
I20260926 12:43:01.767131 32588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:01.771847 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 32507
W20260926 12:43:01.777417 32492 connection.cc:535] client connection to 127.0.0.1:39017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39017): Connection reset by peer (error 104)
2026-09-26T12:43:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:01 dist-test-slave-m3pj krb5kdc[32606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:01 dist-test-slave-m3pj krb5kdc[32606](info): set up 2 sockets
Sep 26 12:43:01 dist-test-slave-m3pj krb5kdc[32606](info): commencing operation
krb5kdc: starting...
W20260926 12:43:02.852154 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:02 dist-test-slave-m3pj krb5kdc[32606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:43:02.983305 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:03.100776 32619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:03.101083 32619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:03.101130 32619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:03.105502 32619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:03.105575 32619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:03.105597 32619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:03.105613 32619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:03.105629 32619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:03.111274 32619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:03.112458 32619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:03.113457 32619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:43:03.122767 32619 server_base.cc:1034] running on GCE node
W20260926 12:43:03.122712 32627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:03.122706 32624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:03.122697 32625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:03.123754 32619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:03.125029 32619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:03.126260 32619 hybrid_clock.cc:648] HybridClock initialized: now 1790426583126198 us; error 76 us; skew 500 ppm
Sep 26 12:43:03 dist-test-slave-m3pj krb5kdc[32606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:03.129448 32619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:43:03.130474 32619 webserver.cc:460] Webserver started at http://127.0.0.1:46109/ using document root <none> and password file <none>
I20260926 12:43:03.131126 32619 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:03.131227 32619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:03.131492 32619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:03.133695 32619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "c7629ee671454547a5d1c7931c8f860f"
format_stamp: "Formatted at 2026-09-26 12:43:03 on dist-test-slave-m3pj"
server_key: "00f69194f66855ce770f2b1e42d99494"
server_key_iv: "ae2388d525bea6eec89aec99a669dcb7"
server_key_version: "encryptionkey@0"
I20260926 12:43:03.134292 32619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "c7629ee671454547a5d1c7931c8f860f"
format_stamp: "Formatted at 2026-09-26 12:43:03 on dist-test-slave-m3pj"
server_key: "00f69194f66855ce770f2b1e42d99494"
server_key_iv: "ae2388d525bea6eec89aec99a669dcb7"
server_key_version: "encryptionkey@0"
I20260926 12:43:03.138545 32619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:43:03.141433 32634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:03.143116 32619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:43:03.143311 32619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "c7629ee671454547a5d1c7931c8f860f"
format_stamp: "Formatted at 2026-09-26 12:43:03 on dist-test-slave-m3pj"
server_key: "00f69194f66855ce770f2b1e42d99494"
server_key_iv: "ae2388d525bea6eec89aec99a669dcb7"
server_key_version: "encryptionkey@0"
I20260926 12:43:03.143458 32619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:03.148680 32619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:03.152110 32619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:03.152335 32619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:03.159483 32619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36153
I20260926 12:43:03.159497 32686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36153 every 8 connection(s)
I20260926 12:43:03.160552 32619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:03.163618 32687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:03.169649 32687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f: Bootstrap starting.
I20260926 12:43:03.170269 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 32619
I20260926 12:43:03.170432 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:03.170725 11138 external_mini_cluster.cc:1428] Setting key 2adcbbbedc427fe45d25013468f3bebe
I20260926 12:43:03.172807 32687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:03.173584 32687 log.cc:824] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:03.176188 32687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f: No bootstrap required, opened a new log
I20260926 12:43:03.176901 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:03.172223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:44055 (local address 127.0.0.1:36153)
0926 12:43:03.172708 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:03.172736 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:03.172774 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:43:03.173582 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:03.173600 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:03.174017 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:03.174453 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:03.175339 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:03.175344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:03.175357 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:03.175368 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:03.175466 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:03.175490 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:03.175667 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:03.175803 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 12:43:03.175999 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":154,"threads_started":1}
I20260926 12:43:03.180433 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:03.180703 32687 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7629ee671454547a5d1c7931c8f860f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } }
I20260926 12:43:03.181020 32687 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:03.181092 32687 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7629ee671454547a5d1c7931c8f860f, State: Initialized, Role: FOLLOWER
I20260926 12:43:03.181608 32687 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7629ee671454547a5d1c7931c8f860f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } }
I20260926 12:43:03.181819 32687 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:03.181957 32687 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:03.182089 32687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:03.183497 32687 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7629ee671454547a5d1c7931c8f860f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } }
I20260926 12:43:03.183944 32687 leader_election.cc:302] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7629ee671454547a5d1c7931c8f860f; no voters:
I20260926 12:43:03.184340 32687 leader_election.cc:288] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:03.184432 32697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:03.184676 32697 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [term 1 LEADER]: Becoming Leader. State: Replica: c7629ee671454547a5d1c7931c8f860f, State: Running, Role: LEADER
I20260926 12:43:03.184875 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:03.182930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:38229 (local address 127.0.0.1:36153)
0926 12:43:03.183068 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:03.183073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:03.183228 (+ 155us) server_negotiation.cc:407] Connection header received
0926 12:43:03.183371 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:03.183374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:03.183416 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:03.183491 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:03.184280 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:03.184283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:03.184286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:03.184320 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:03.184354 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:03.184369 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:03.184468 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:03.184703 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 12:43:03.184755 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:43:03.185310 32697 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7629ee671454547a5d1c7931c8f860f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } }
I20260926 12:43:03.185747 32687 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:03.186635 32697 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7629ee671454547a5d1c7931c8f860f. Latest consensus state: current_term: 1 leader_uuid: "c7629ee671454547a5d1c7931c8f860f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7629ee671454547a5d1c7931c8f860f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } } }
I20260926 12:43:03.186764 32697 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:03.187160 32706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:03.187448 32698 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7629ee671454547a5d1c7931c8f860f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7629ee671454547a5d1c7931c8f860f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36153 } } }
I20260926 12:43:03.187620 32698 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:03.190768 32706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:03.197300 32706 catalog_manager.cc:1347] Generated new cluster ID: c4d9396120014daa84a8d5043c627f29
I20260926 12:43:03.197396 32706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:03.207069 32706 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:03.207445 32706 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:03.214133 32706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7629ee671454547a5d1c7931c8f860f: Generated new TSK 0
I20260926 12:43:03.215056 32706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:03.219287 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 32619
2026-09-26T12:43:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:03 dist-test-slave-m3pj krb5kdc[32718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:03 dist-test-slave-m3pj krb5kdc[32718](info): set up 2 sockets
Sep 26 12:43:03 dist-test-slave-m3pj krb5kdc[32718](info): commencing operation
krb5kdc: starting...
W20260926 12:43:04.308234 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:04 dist-test-slave-m3pj krb5kdc[32718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:43:04.439575 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:44441
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:44441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:04.554096 32731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:04.554459 32731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:04.554572 32731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:04.558982 32731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:04.559090 32731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:04.559129 32731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:04.559168 32731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:04.559202 32731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:04.564622 32731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:44441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:04.565725 32731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:04.566781 32731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:04.576184 32736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:04.576201 32737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:04.576325 32731 server_base.cc:1034] running on GCE node
W20260926 12:43:04.576198 32739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:04.577071 32731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:04.578222 32731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:04.579427 32731 hybrid_clock.cc:648] HybridClock initialized: now 1790426584579394 us; error 41 us; skew 500 ppm
Sep 26 12:43:04 dist-test-slave-m3pj krb5kdc[32718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426584, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:04.582441 32731 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:43:04.583459 32731 webserver.cc:460] Webserver started at http://127.10.224.190:45235/ using document root <none> and password file <none>
I20260926 12:43:04.584085 32731 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:04.584151 32731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:04.584323 32731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:04.586447 32731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "70f628708fff4988a6c65a4e49a3445d"
format_stamp: "Formatted at 2026-09-26 12:43:04 on dist-test-slave-m3pj"
server_key: "cf18e994f51eab88e612eda05e3a1109"
server_key_iv: "ee56434d8a4d92581886e7390ba0d163"
server_key_version: "encryptionkey@0"
I20260926 12:43:04.587015 32731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "70f628708fff4988a6c65a4e49a3445d"
format_stamp: "Formatted at 2026-09-26 12:43:04 on dist-test-slave-m3pj"
server_key: "cf18e994f51eab88e612eda05e3a1109"
server_key_iv: "ee56434d8a4d92581886e7390ba0d163"
server_key_version: "encryptionkey@0"
I20260926 12:43:04.591073 32731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:43:04.593725 32746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:04.595186 32731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:43:04.595353 32731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "70f628708fff4988a6c65a4e49a3445d"
format_stamp: "Formatted at 2026-09-26 12:43:04 on dist-test-slave-m3pj"
server_key: "cf18e994f51eab88e612eda05e3a1109"
server_key_iv: "ee56434d8a4d92581886e7390ba0d163"
server_key_version: "encryptionkey@0"
I20260926 12:43:04.595484 32731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:04.613572 32731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:04.617265 32731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:04.617517 32731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:04.625298 32731 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:44441
I20260926 12:43:04.625324 330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:44441 every 8 connection(s)
I20260926 12:43:04.626351 32731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:04.629309 331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:04.635115 331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d: Bootstrap starting.
I20260926 12:43:04.636634 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 32731
I20260926 12:43:04.636763 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:04.637108 11138 external_mini_cluster.cc:1428] Setting key e532c3bedf3481a2cc38c78a74103b23
I20260926 12:43:04.638091 331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:04.639015 331 log.cc:824] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:04.641175 331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d: No bootstrap required, opened a new log
I20260926 12:43:04.645051 331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f628708fff4988a6c65a4e49a3445d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44441 } }
I20260926 12:43:04.645308 331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:04.645344 331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f628708fff4988a6c65a4e49a3445d, State: Initialized, Role: FOLLOWER
I20260926 12:43:04.645890 331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f628708fff4988a6c65a4e49a3445d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44441 } }
I20260926 12:43:04.646082 331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:04.646170 331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:04.646307 331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 0 FOLLOWER]: Advancing to term 1
Sep 26 12:43:04 dist-test-slave-m3pj krb5kdc[32718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:43:04.647382 331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f628708fff4988a6c65a4e49a3445d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44441 } }
I20260926 12:43:04.647855 331 leader_election.cc:302] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f628708fff4988a6c65a4e49a3445d; no voters:
I20260926 12:43:04.648329 331 leader_election.cc:288] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:04.648440 336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:04.648728 336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [term 1 LEADER]: Becoming Leader. State: Replica: 70f628708fff4988a6c65a4e49a3445d, State: Running, Role: LEADER
I20260926 12:43:04.649151 336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f628708fff4988a6c65a4e49a3445d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44441 } }
I20260926 12:43:04.649821 331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:04.650977 337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f628708fff4988a6c65a4e49a3445d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f628708fff4988a6c65a4e49a3445d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44441 } } }
I20260926 12:43:04.651096 337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:04.651324 337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f628708fff4988a6c65a4e49a3445d. Latest consensus state: current_term: 1 leader_uuid: "70f628708fff4988a6c65a4e49a3445d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f628708fff4988a6c65a4e49a3445d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 44441 } } }
I20260926 12:43:04.651393 337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:04.654295 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:04.638364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49333 (local address 127.10.224.190:44441)
0926 12:43:04.638835 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:04.638851 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:04.638887 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:43:04.639899 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:04.639936 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:04.640303 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:04.640656 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:04.641561 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:04.642572 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:04.643483 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:04.644076 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:04.644254 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:04.647276 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:04.647282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:04.647298 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:04.647330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:04.649570 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:04.650255 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:04.650263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:04.650272 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:04.650437 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:04.650921 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:04.650924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:04.650926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:04.651590 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:04.652598 (+ 1008us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:04.652799 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 12:43:04.653083 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":314,"thread_start_us":133,"threads_started":1}
W20260926 12:43:04.655432 351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:43:04.655563 351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:43:04.655896 345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:04.656632 345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:04.658100 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:04.663453 345 catalog_manager.cc:1347] Generated new cluster ID: f45310f1a5fa4530b865f507b9e691d3
I20260926 12:43:04.663582 345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:04.669147 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:04.660367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40587 (local address 127.10.224.190:44441)
0926 12:43:04.660492 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:04.660498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:04.660776 (+ 278us) server_negotiation.cc:407] Connection header received
0926 12:43:04.661373 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:04.661375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:04.661420 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:04.661525 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:04.662227 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:04.662788 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:04.663701 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:04.663926 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:04.664045 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:04.665605 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:04.665609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:04.665611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:04.665636 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:04.667127 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:04.667666 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:04.667676 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:04.667677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:04.667754 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:04.668104 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:04.668108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:04.668110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:04.668295 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:04.668456 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:04.668731 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 12:43:04.668832 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:43:04.681880 345 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:04.683223 345 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:04.688270 345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f628708fff4988a6c65a4e49a3445d: Generated new TSK 0
I20260926 12:43:04.689010 345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:04.696552 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 32731
W20260926 12:43:04.702280 32716 connection.cc:535] client connection to 127.10.224.190:44441 recv error: Network error: failed to read from TLS socket (remote: 127.10.224.190:44441): Connection reset by peer (error 104)
2026-09-26T12:43:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:04 dist-test-slave-m3pj krb5kdc[362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:04 dist-test-slave-m3pj krb5kdc[362](info): set up 2 sockets
Sep 26 12:43:04 dist-test-slave-m3pj krb5kdc[362](info): commencing operation
krb5kdc: starting...
W20260926 12:43:05.748651 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:05 dist-test-slave-m3pj krb5kdc[362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:43:05.880374 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:37185
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:37185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:06.001077 375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:06.001379 375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:06.001425 375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:06.005656 375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:06.005722 375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:06.005743 375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:06.005759 375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:06.005774 375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:06.011391 375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:37185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:37185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:06.012516 375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:06.013494 375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:06.022817 380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:06.022886 383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:06.022859 381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:06.023072 375 server_base.cc:1034] running on GCE node
I20260926 12:43:06.023895 375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:06.025203 375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:06.026477 375 hybrid_clock.cc:648] HybridClock initialized: now 1790426586026464 us; error 36 us; skew 500 ppm
Sep 26 12:43:06 dist-test-slave-m3pj krb5kdc[362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426586, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:06.034286 375 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:43:06.035317 375 webserver.cc:460] Webserver started at http://127.10.224.190:43147/ using document root <none> and password file <none>
I20260926 12:43:06.035972 375 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:06.036042 375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:06.036217 375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:06.038393 375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "a475be9c41c94bff9a69e30324d4c643"
format_stamp: "Formatted at 2026-09-26 12:43:06 on dist-test-slave-m3pj"
server_key: "faeff30f5f0def1987f849155c4ab042"
server_key_iv: "c64bd0d7420acacf348c829381ec222c"
server_key_version: "encryptionkey@0"
I20260926 12:43:06.038913 375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "a475be9c41c94bff9a69e30324d4c643"
format_stamp: "Formatted at 2026-09-26 12:43:06 on dist-test-slave-m3pj"
server_key: "faeff30f5f0def1987f849155c4ab042"
server_key_iv: "c64bd0d7420acacf348c829381ec222c"
server_key_version: "encryptionkey@0"
I20260926 12:43:06.042912 375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:43:06.045552 390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:06.047109 375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:43:06.047291 375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "a475be9c41c94bff9a69e30324d4c643"
format_stamp: "Formatted at 2026-09-26 12:43:06 on dist-test-slave-m3pj"
server_key: "faeff30f5f0def1987f849155c4ab042"
server_key_iv: "c64bd0d7420acacf348c829381ec222c"
server_key_version: "encryptionkey@0"
I20260926 12:43:06.047429 375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:06.064690 375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:06.069530 375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:06.069780 375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:06.081207 375 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:37185
I20260926 12:43:06.081615 443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:37185 every 8 connection(s)
I20260926 12:43:06.082374 375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:06.086302 444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:06.088124 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 375
I20260926 12:43:06.088299 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:06.088627 11138 external_mini_cluster.cc:1428] Setting key d0c5d9257527c533add2633f76609a68
I20260926 12:43:06.094403 444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643: Bootstrap starting.
I20260926 12:43:06.097239 444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:06.098237 444 log.cc:824] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:06.100481 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:06.090696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44579 (local address 127.10.224.190:37185)
0926 12:43:06.091180 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:06.091201 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:06.091247 (+ 46us) server_negotiation.cc:407] Connection header received
0926 12:43:06.092133 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:06.092160 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:06.092642 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:06.093036 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:06.093484 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:06.094897 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:06.096289 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:06.096975 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:06.097168 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:06.098387 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:06.098394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:06.098412 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:06.098969 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:06.099031 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:06.099057 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:06.099231 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:06.099422 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 12:43:06.099630 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":125,"threads_started":1}
I20260926 12:43:06.101747 444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643: No bootstrap required, opened a new log
I20260926 12:43:06.104596 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:06.108160 444 raft_consensus.cc:348] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a475be9c41c94bff9a69e30324d4c643" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 37185 } }
I20260926 12:43:06.108558 444 raft_consensus.cc:374] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:06.108633 444 raft_consensus.cc:729] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a475be9c41c94bff9a69e30324d4c643, State: Initialized, Role: FOLLOWER
I20260926 12:43:06.109236 444 consensus_queue.cc:260] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a475be9c41c94bff9a69e30324d4c643" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 37185 } }
I20260926 12:43:06.109462 444 raft_consensus.cc:388] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:06.109558 444 raft_consensus.cc:482] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:06.109691 444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:06.111315 444 raft_consensus.cc:504] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a475be9c41c94bff9a69e30324d4c643" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 37185 } }
I20260926 12:43:06.111845 444 leader_election.cc:302] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a475be9c41c94bff9a69e30324d4c643; no voters:
I20260926 12:43:06.112011 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:06.107007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52895 (local address 127.10.224.190:37185)
0926 12:43:06.107136 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:06.107141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:06.107397 (+ 256us) server_negotiation.cc:407] Connection header received
0926 12:43:06.107596 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:06.107601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:06.107658 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:06.107743 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:06.108053 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:06.108930 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:06.109981 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:06.110393 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:06.110498 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:06.111431 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:06.111438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:06.111440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:06.111457 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:06.111497 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:06.111513 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:06.111639 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:06.111759 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 12:43:06.111794 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:43:06.112315 444 leader_election.cc:288] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:06.112458 454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:06.112723 454 raft_consensus.cc:686] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [term 1 LEADER]: Becoming Leader. State: Replica: a475be9c41c94bff9a69e30324d4c643, State: Running, Role: LEADER
I20260926 12:43:06.113131 454 consensus_queue.cc:237] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a475be9c41c94bff9a69e30324d4c643" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 37185 } }
I20260926 12:43:06.113951 444 sys_catalog.cc:564] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:06.114902 455 sys_catalog.cc:455] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a475be9c41c94bff9a69e30324d4c643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a475be9c41c94bff9a69e30324d4c643" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 37185 } } }
I20260926 12:43:06.115046 455 sys_catalog.cc:458] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:06.115316 456 sys_catalog.cc:455] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a475be9c41c94bff9a69e30324d4c643. Latest consensus state: current_term: 1 leader_uuid: "a475be9c41c94bff9a69e30324d4c643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a475be9c41c94bff9a69e30324d4c643" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 37185 } } }
I20260926 12:43:06.115406 456 sys_catalog.cc:458] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:06.115592 463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:06.119118 463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:06.125452 463 catalog_manager.cc:1347] Generated new cluster ID: a6ddda5d18d946fcb8312b673a71f2f3
I20260926 12:43:06.125553 463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:06.134374 463 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:06.135743 463 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:06.161540 463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a475be9c41c94bff9a69e30324d4c643: Generated new TSK 0
I20260926 12:43:06.162523 463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:06.174772 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 375
W20260926 12:43:06.180779 360 connection.cc:535] client connection to 127.10.224.190:37185 recv error: Network error: failed to read from TLS socket (remote: 127.10.224.190:37185): Connection reset by peer (error 104)
2026-09-26T12:43:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:06 dist-test-slave-m3pj krb5kdc[475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:06 dist-test-slave-m3pj krb5kdc[475](info): set up 2 sockets
Sep 26 12:43:06 dist-test-slave-m3pj krb5kdc[475](info): commencing operation
krb5kdc: starting...
W20260926 12:43:07.232281 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:07 dist-test-slave-m3pj krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:43:07.363308 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39613
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:07.483986 488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:07.484344 488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:07.484447 488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:07.488618 488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:07.488729 488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:07.488770 488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:07.488814 488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:07.488842 488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:07.494383 488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:39613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:07.495536 488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:07.496549 488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:07.505571 496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:07.505527 493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:07.505527 494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:07.505625 488 server_base.cc:1034] running on GCE node
I20260926 12:43:07.506400 488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:07.507668 488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:07.508877 488 hybrid_clock.cc:648] HybridClock initialized: now 1790426587508843 us; error 48 us; skew 500 ppm
Sep 26 12:43:07 dist-test-slave-m3pj krb5kdc[475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426587, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:07.518129 488 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:43:07.519166 488 webserver.cc:460] Webserver started at http://127.10.224.190:44471/ using document root <none> and password file <none>
I20260926 12:43:07.519882 488 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:07.519977 488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:07.520236 488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:07.522429 488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "241593e6805146adaff0788d85600bf7"
format_stamp: "Formatted at 2026-09-26 12:43:07 on dist-test-slave-m3pj"
server_key: "6357abe92eadb936491db9113865ae41"
server_key_iv: "7794be00f8950f8daa5454ef9ae64c89"
server_key_version: "encryptionkey@0"
I20260926 12:43:07.523048 488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "241593e6805146adaff0788d85600bf7"
format_stamp: "Formatted at 2026-09-26 12:43:07 on dist-test-slave-m3pj"
server_key: "6357abe92eadb936491db9113865ae41"
server_key_iv: "7794be00f8950f8daa5454ef9ae64c89"
server_key_version: "encryptionkey@0"
I20260926 12:43:07.527081 488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:43:07.529748 503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:07.531055 488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:43:07.531225 488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "241593e6805146adaff0788d85600bf7"
format_stamp: "Formatted at 2026-09-26 12:43:07 on dist-test-slave-m3pj"
server_key: "6357abe92eadb936491db9113865ae41"
server_key_iv: "7794be00f8950f8daa5454ef9ae64c89"
server_key_version: "encryptionkey@0"
I20260926 12:43:07.531358 488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:07.536248 488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:07.539314 488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:07.539588 488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:07.547103 488 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:39613
I20260926 12:43:07.547128 555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:39613 every 8 connection(s)
I20260926 12:43:07.548172 488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:07.550081 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 488
I20260926 12:43:07.550241 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:07.550556 11138 external_mini_cluster.cc:1428] Setting key 497d81c30487931c6337933b124f846b
I20260926 12:43:07.551335 556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:07.556007 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:07.552057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40915 (local address 127.10.224.190:39613)
0926 12:43:07.552464 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:07.552490 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:07.552527 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:43:07.553182 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:07.553197 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:07.553478 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:07.553758 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:07.554640 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:07.554645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:07.554657 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:07.554668 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:07.554722 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:07.554742 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:07.554853 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:07.555014 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 12:43:07.555205 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":134,"threads_started":1}
I20260926 12:43:07.558190 556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7: Bootstrap starting.
I20260926 12:43:07.558882 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:07.560667 556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:07.561573 556 log.cc:824] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:07.562827 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:07.561025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41185 (local address 127.10.224.190:39613)
0926 12:43:07.561151 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:07.561157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:07.561300 (+ 143us) server_negotiation.cc:407] Connection header received
0926 12:43:07.561502 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:07.561505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:07.561544 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:07.561629 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:07.562378 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:07.562381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:07.562383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:07.562393 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:07.562417 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:07.562427 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:07.562535 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:07.562662 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 12:43:07.562700 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:43:07.563933 556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7: No bootstrap required, opened a new log
I20260926 12:43:07.567404 556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241593e6805146adaff0788d85600bf7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39613 } }
I20260926 12:43:07.567694 556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:07.567773 556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241593e6805146adaff0788d85600bf7, State: Initialized, Role: FOLLOWER
I20260926 12:43:07.568246 556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241593e6805146adaff0788d85600bf7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39613 } }
I20260926 12:43:07.568418 556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:07.568499 556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:07.568606 556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:07.569631 556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241593e6805146adaff0788d85600bf7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39613 } }
I20260926 12:43:07.569929 556 leader_election.cc:302] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241593e6805146adaff0788d85600bf7; no voters:
I20260926 12:43:07.570187 556 leader_election.cc:288] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:07.570312 566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:07.570623 566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [term 1 LEADER]: Becoming Leader. State: Replica: 241593e6805146adaff0788d85600bf7, State: Running, Role: LEADER
I20260926 12:43:07.570942 566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241593e6805146adaff0788d85600bf7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39613 } }
I20260926 12:43:07.571506 556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:07.572671 568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241593e6805146adaff0788d85600bf7. Latest consensus state: current_term: 1 leader_uuid: "241593e6805146adaff0788d85600bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241593e6805146adaff0788d85600bf7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39613 } } }
I20260926 12:43:07.572803 568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:07.573086 567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241593e6805146adaff0788d85600bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241593e6805146adaff0788d85600bf7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39613 } } }
I20260926 12:43:07.573179 567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:07.573603 575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:07.576435 575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:07.582386 575 catalog_manager.cc:1347] Generated new cluster ID: f0c88a0eacec4d80a482e8c1457c0caa
I20260926 12:43:07.582533 575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:07.593847 575 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:07.594246 575 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:07.600382 575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241593e6805146adaff0788d85600bf7: Generated new TSK 0
I20260926 12:43:07.601253 575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:07.608227 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 488
2026-09-26T12:43:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:07 dist-test-slave-m3pj krb5kdc[587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:07 dist-test-slave-m3pj krb5kdc[587](info): set up 2 sockets
Sep 26 12:43:07 dist-test-slave-m3pj krb5kdc[587](info): commencing operation
krb5kdc: starting...
W20260926 12:43:08.660395 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:08 dist-test-slave-m3pj krb5kdc[587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:43:08.787765 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39497
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:39497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:08.902274 600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:08.902674 600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:08.902766 600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:08.907204 600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:08.907328 600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:08.907366 600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:08.907407 600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:08.907438 600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:08.913190 600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:39497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:08.914403 600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:08.915549 600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:43:08.924232 600 server_base.cc:1034] running on GCE node
W20260926 12:43:08.924191 605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:08.924170 606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:08.924172 608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:08.925122 600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:08.926240 600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:08.927445 600 hybrid_clock.cc:648] HybridClock initialized: now 1790426588927405 us; error 50 us; skew 500 ppm
Sep 26 12:43:08 dist-test-slave-m3pj krb5kdc[587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426588, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:08.930913 600 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:43:08.931965 600 webserver.cc:460] Webserver started at http://127.10.224.190:35783/ using document root <none> and password file <none>
I20260926 12:43:08.932538 600 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:08.932605 600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:08.932787 600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:08.934949 600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "b88a27d3bb554a2095e64f9e32614e03"
format_stamp: "Formatted at 2026-09-26 12:43:08 on dist-test-slave-m3pj"
server_key: "a8c9ecbb162441cbc9ef0063fc2c09b3"
server_key_iv: "788e76376cef96f93c549f323ef783ff"
server_key_version: "encryptionkey@0"
I20260926 12:43:08.935499 600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "b88a27d3bb554a2095e64f9e32614e03"
format_stamp: "Formatted at 2026-09-26 12:43:08 on dist-test-slave-m3pj"
server_key: "a8c9ecbb162441cbc9ef0063fc2c09b3"
server_key_iv: "788e76376cef96f93c549f323ef783ff"
server_key_version: "encryptionkey@0"
I20260926 12:43:08.939651 600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:43:08.942346 615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:08.943920 600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:43:08.944092 600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "b88a27d3bb554a2095e64f9e32614e03"
format_stamp: "Formatted at 2026-09-26 12:43:08 on dist-test-slave-m3pj"
server_key: "a8c9ecbb162441cbc9ef0063fc2c09b3"
server_key_iv: "788e76376cef96f93c549f323ef783ff"
server_key_version: "encryptionkey@0"
I20260926 12:43:08.944226 600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:08.957094 600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:08.960574 600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:08.960806 600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:08.968097 600 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:39497
I20260926 12:43:08.968127 667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:39497 every 8 connection(s)
I20260926 12:43:08.969142 600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:08.972072 668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:08.974807 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 600
I20260926 12:43:08.974987 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:08.975327 11138 external_mini_cluster.cc:1428] Setting key 82e3c6913c0e6be1e3c52a49d6062399
I20260926 12:43:08.978791 668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03: Bootstrap starting.
I20260926 12:43:08.981326 668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:08.982162 668 log.cc:824] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:08.984391 668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03: No bootstrap required, opened a new log
Sep 26 12:43:08 dist-test-slave-m3pj krb5kdc[587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:43:08.988363 668 raft_consensus.cc:348] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a27d3bb554a2095e64f9e32614e03" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39497 } }
I20260926 12:43:08.988646 668 raft_consensus.cc:374] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:08.988747 668 raft_consensus.cc:729] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88a27d3bb554a2095e64f9e32614e03, State: Initialized, Role: FOLLOWER
I20260926 12:43:08.989283 668 consensus_queue.cc:260] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a27d3bb554a2095e64f9e32614e03" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39497 } }
I20260926 12:43:08.989421 668 raft_consensus.cc:388] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:08.989588 668 raft_consensus.cc:482] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:08.989719 668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:08.990756 668 raft_consensus.cc:504] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a27d3bb554a2095e64f9e32614e03" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39497 } }
I20260926 12:43:08.991171 668 leader_election.cc:302] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b88a27d3bb554a2095e64f9e32614e03; no voters:
I20260926 12:43:08.991565 668 leader_election.cc:288] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:08.991671 673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:08.991918 673 raft_consensus.cc:686] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [term 1 LEADER]: Becoming Leader. State: Replica: b88a27d3bb554a2095e64f9e32614e03, State: Running, Role: LEADER
I20260926 12:43:08.992053 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:08.976892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47987 (local address 127.10.224.190:39497)
0926 12:43:08.977408 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:08.977423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:08.977492 (+ 69us) server_negotiation.cc:407] Connection header received
0926 12:43:08.978144 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:08.978161 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:08.978551 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:08.978961 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:08.979883 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:08.980917 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:08.981902 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:08.982483 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:08.982673 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:08.985527 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:08.985534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:08.985550 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:08.985584 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:08.988436 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:08.989059 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:08.989073 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:08.989079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:08.989209 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:08.989529 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:08.989534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:08.989537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:08.990054 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:08.990267 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:08.990585 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 12:43:08.990956 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":167,"threads_started":1}
I20260926 12:43:08.992308 673 consensus_queue.cc:237] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a27d3bb554a2095e64f9e32614e03" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39497 } }
I20260926 12:43:08.992964 668 sys_catalog.cc:564] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:08.994004 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b88a27d3bb554a2095e64f9e32614e03. Latest consensus state: current_term: 1 leader_uuid: "b88a27d3bb554a2095e64f9e32614e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a27d3bb554a2095e64f9e32614e03" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39497 } } }
I20260926 12:43:08.994135 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:08.994477 674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b88a27d3bb554a2095e64f9e32614e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88a27d3bb554a2095e64f9e32614e03" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39497 } } }
I20260926 12:43:08.994585 674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:08.996047 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 12:43:08.998186 688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:43:08.998270 688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:43:08.998347 682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:08.999226 682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:09.006326 682 catalog_manager.cc:1347] Generated new cluster ID: ad4a77e00a664cc5aac361ec22c78f41
I20260926 12:43:09.006498 682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:09.006841 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:08.998054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37731 (local address 127.10.224.190:39497)
0926 12:43:08.998171 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:08.998176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:08.998349 (+ 173us) server_negotiation.cc:407] Connection header received
0926 12:43:08.998523 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:08.998526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:08.998582 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:08.998673 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:08.999418 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:09.000084 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:09.000947 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:09.001220 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:09.001340 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:09.002790 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:09.002794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:09.002797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:09.002825 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:09.004811 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:09.005337 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:09.005340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:09.005342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:09.005438 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:09.005744 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:09.005747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:09.005749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:09.005984 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:09.006124 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:09.006575 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 12:43:09.006701 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 12:43:09.027807 682 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:09.029207 682 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:09.037359 682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b88a27d3bb554a2095e64f9e32614e03: Generated new TSK 0
I20260926 12:43:09.038266 682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:09.049417 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 600
2026-09-26T12:43:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:09 dist-test-slave-m3pj krb5kdc[699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:09 dist-test-slave-m3pj krb5kdc[699](info): set up 2 sockets
Sep 26 12:43:09 dist-test-slave-m3pj krb5kdc[699](info): commencing operation
krb5kdc: starting...
W20260926 12:43:10.112361 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:10 dist-test-slave-m3pj krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:43:10.248566 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39113
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:39113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:10.374864 712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:10.375241 712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:10.375351 712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:10.380175 712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:10.380308 712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:10.380350 712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:10.380396 712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:10.380432 712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:10.386917 712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:39113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:10.388269 712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:10.389528 712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:10.400215 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:10.400260 712 server_base.cc:1034] running on GCE node
W20260926 12:43:10.400215 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:10.400233 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:10.401060 712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:10.402437 712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:10.403635 712 hybrid_clock.cc:648] HybridClock initialized: now 1790426590403620 us; error 32 us; skew 500 ppm
Sep 26 12:43:10 dist-test-slave-m3pj krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426590, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:10.406708 712 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:43:10.407744 712 webserver.cc:460] Webserver started at http://127.10.224.190:41709/ using document root <none> and password file <none>
I20260926 12:43:10.408339 712 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:10.408433 712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:10.408680 712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:10.410830 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "1d7509071b824e64b0aa18a4e43964b1"
format_stamp: "Formatted at 2026-09-26 12:43:10 on dist-test-slave-m3pj"
server_key: "ead496ba3dfde856a24b32b48c22a166"
server_key_iv: "17ced1348d339509090578962f9dcf15"
server_key_version: "encryptionkey@0"
I20260926 12:43:10.411417 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "1d7509071b824e64b0aa18a4e43964b1"
format_stamp: "Formatted at 2026-09-26 12:43:10 on dist-test-slave-m3pj"
server_key: "ead496ba3dfde856a24b32b48c22a166"
server_key_iv: "17ced1348d339509090578962f9dcf15"
server_key_version: "encryptionkey@0"
I20260926 12:43:10.416046 712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:43:10.418730 727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:10.420253 712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:43:10.420423 712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "1d7509071b824e64b0aa18a4e43964b1"
format_stamp: "Formatted at 2026-09-26 12:43:10 on dist-test-slave-m3pj"
server_key: "ead496ba3dfde856a24b32b48c22a166"
server_key_iv: "17ced1348d339509090578962f9dcf15"
server_key_version: "encryptionkey@0"
I20260926 12:43:10.420562 712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:10.439472 712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:10.443250 712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:10.443491 712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:10.450639 712 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:39113
I20260926 12:43:10.450711 779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:39113 every 8 connection(s)
I20260926 12:43:10.451787 712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:10.454867 780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:10.456267 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 712
I20260926 12:43:10.456429 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:10.456784 11138 external_mini_cluster.cc:1428] Setting key c0febc9017d7c27c8861189ea6088b4c
I20260926 12:43:10.462240 780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1: Bootstrap starting.
I20260926 12:43:10.465457 780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:10.466600 780 log.cc:824] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:10.468204 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:10.458295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57659 (local address 127.10.224.190:39113)
0926 12:43:10.458753 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:10.458772 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:10.458817 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:43:10.459709 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:10.459737 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:10.460220 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:10.460640 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:10.461116 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:10.462624 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:10.463904 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:10.464663 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:10.464842 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:10.465896 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:10.465905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:10.465927 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:10.466492 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:10.466585 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:10.466621 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:10.466735 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:10.466926 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 12:43:10.467183 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260926 12:43:10.469142 780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1: No bootstrap required, opened a new log
I20260926 12:43:10.471822 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:10.474188 780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7509071b824e64b0aa18a4e43964b1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39113 } }
I20260926 12:43:10.474577 780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:10.474658 780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7509071b824e64b0aa18a4e43964b1, State: Initialized, Role: FOLLOWER
I20260926 12:43:10.475232 780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7509071b824e64b0aa18a4e43964b1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39113 } }
I20260926 12:43:10.475445 780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:10.475555 780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:10.475669 780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:10.476976 780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7509071b824e64b0aa18a4e43964b1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39113 } }
I20260926 12:43:10.477445 780 leader_election.cc:302] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d7509071b824e64b0aa18a4e43964b1; no voters:
I20260926 12:43:10.477836 780 leader_election.cc:288] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:10.477957 790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:10.478217 790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [term 1 LEADER]: Becoming Leader. State: Replica: 1d7509071b824e64b0aa18a4e43964b1, State: Running, Role: LEADER
I20260926 12:43:10.478639 790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7509071b824e64b0aa18a4e43964b1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39113 } }
I20260926 12:43:10.479060 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:10.474333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41973 (local address 127.10.224.190:39113)
0926 12:43:10.474485 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:10.474489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:10.474717 (+ 228us) server_negotiation.cc:407] Connection header received
0926 12:43:10.474834 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:10.474836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:10.474883 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:10.474994 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:10.475299 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:10.476054 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:10.477225 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:10.477599 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:10.477708 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:10.478621 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:10.478625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:10.478627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:10.478642 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:10.478677 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:10.478691 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:10.478770 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:10.478868 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 12:43:10.478927 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:43:10.479444 780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:10.480522 791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d7509071b824e64b0aa18a4e43964b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7509071b824e64b0aa18a4e43964b1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39113 } } }
I20260926 12:43:10.480695 791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:10.480932 791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d7509071b824e64b0aa18a4e43964b1. Latest consensus state: current_term: 1 leader_uuid: "1d7509071b824e64b0aa18a4e43964b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7509071b824e64b0aa18a4e43964b1" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 39113 } } }
I20260926 12:43:10.481081 791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:10.481106 799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:10.484767 799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:10.491236 799 catalog_manager.cc:1347] Generated new cluster ID: e273732d74c849e68a9f3d287d0f1603
I20260926 12:43:10.491358 799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:10.498356 799 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:10.499753 799 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:10.506350 799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d7509071b824e64b0aa18a4e43964b1: Generated new TSK 0
I20260926 12:43:10.507157 799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:10.517148 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 712
2026-09-26T12:43:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:10 dist-test-slave-m3pj krb5kdc[811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:10 dist-test-slave-m3pj krb5kdc[811](info): set up 2 sockets
Sep 26 12:43:10 dist-test-slave-m3pj krb5kdc[811](info): commencing operation
krb5kdc: starting...
W20260926 12:43:11.568451 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:11 dist-test-slave-m3pj krb5kdc[811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:43:11.703732 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:36991
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:36991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:11.826972 824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:11.827291 824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:11.827337 824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:11.831627 824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:11.831696 824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:11.831717 824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:11.831734 824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:11.831750 824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:11.837247 824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:36991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:36991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:11.838357 824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:11.839396 824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:11.848788 830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:11.848793 829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:11.849090 824 server_base.cc:1034] running on GCE node
W20260926 12:43:11.848800 832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:11.849807 824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:11.851104 824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:11.852283 824 hybrid_clock.cc:648] HybridClock initialized: now 1790426591852266 us; error 30 us; skew 500 ppm
Sep 26 12:43:11 dist-test-slave-m3pj krb5kdc[811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426591, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:11.855314 824 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:43:11.856388 824 webserver.cc:460] Webserver started at http://127.10.224.190:39283/ using document root <none> and password file <none>
I20260926 12:43:11.856966 824 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:11.857028 824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:11.857206 824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:11.859378 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d"
format_stamp: "Formatted at 2026-09-26 12:43:11 on dist-test-slave-m3pj"
server_key: "18ffcfbc752775aa8d1db61bd160b2cc"
server_key_iv: "086e5db272dfe8d72fb737f4debaaf1e"
server_key_version: "encryptionkey@0"
I20260926 12:43:11.860044 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d"
format_stamp: "Formatted at 2026-09-26 12:43:11 on dist-test-slave-m3pj"
server_key: "18ffcfbc752775aa8d1db61bd160b2cc"
server_key_iv: "086e5db272dfe8d72fb737f4debaaf1e"
server_key_version: "encryptionkey@0"
I20260926 12:43:11.864342 824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:43:11.867180 839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:11.868647 824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:43:11.868894 824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d"
format_stamp: "Formatted at 2026-09-26 12:43:11 on dist-test-slave-m3pj"
server_key: "18ffcfbc752775aa8d1db61bd160b2cc"
server_key_iv: "086e5db272dfe8d72fb737f4debaaf1e"
server_key_version: "encryptionkey@0"
I20260926 12:43:11.869027 824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:11.874154 824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:11.877938 824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:11.878163 824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:11.886082 824 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:36991
I20260926 12:43:11.886097 891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:36991 every 8 connection(s)
I20260926 12:43:11.887090 824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:11.890015 892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:11.891319 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 824
I20260926 12:43:11.891570 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:11.891865 11138 external_mini_cluster.cc:1428] Setting key 32d5e5965f0d5f80a7379c31fb4a98e6
I20260926 12:43:11.896675 892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d: Bootstrap starting.
I20260926 12:43:11.897162 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:11.893266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47849 (local address 127.10.224.190:36991)
0926 12:43:11.893725 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:11.893741 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:11.893776 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:43:11.894426 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:11.894444 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:11.894739 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:11.895103 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:11.895903 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:11.895908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:11.895920 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:11.895931 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:11.895985 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:11.896011 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:11.896165 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:11.896239 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 12:43:11.896401 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":144,"threads_started":1}
I20260926 12:43:11.899313 892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:11.900206 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:11.900321 892 log.cc:824] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:11.902698 892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d: No bootstrap required, opened a new log
I20260926 12:43:11.904402 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:11.902313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48787 (local address 127.10.224.190:36991)
0926 12:43:11.902447 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:11.902451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:11.902829 (+ 378us) server_negotiation.cc:407] Connection header received
0926 12:43:11.903064 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:11.903067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:11.903113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:11.903210 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:11.903971 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:11.903975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:11.903978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:43:11.903989 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:11.904016 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:43:11.904036 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:11.904130 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:11.904231 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 12:43:11.904271 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:43:11.906932 892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36991 } }
I20260926 12:43:11.907205 892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:43:11.907274 892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f87df05c8204d94a3b1d8b8f37a9d0d, State: Initialized, Role: FOLLOWER
I20260926 12:43:11.907810 892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36991 } }
I20260926 12:43:11.907974 892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:43:11.908044 892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:43:11.908170 892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:11.909204 892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36991 } }
I20260926 12:43:11.909587 892 leader_election.cc:302] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f87df05c8204d94a3b1d8b8f37a9d0d; no voters:
I20260926 12:43:11.909931 892 leader_election.cc:288] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:43:11.910094 902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:11.910353 902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [term 1 LEADER]: Becoming Leader. State: Replica: 5f87df05c8204d94a3b1d8b8f37a9d0d, State: Running, Role: LEADER
I20260926 12:43:11.910801 902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36991 } }
I20260926 12:43:11.911407 892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:11.912782 904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f87df05c8204d94a3b1d8b8f37a9d0d. Latest consensus state: current_term: 1 leader_uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36991 } } }
I20260926 12:43:11.912844 903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f87df05c8204d94a3b1d8b8f37a9d0d" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 36991 } } }
I20260926 12:43:11.912909 904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:11.912940 903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:11.913298 911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:11.916265 911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:11.922348 911 catalog_manager.cc:1347] Generated new cluster ID: 0b663b0459854c4589693dda9882a0f2
I20260926 12:43:11.922444 911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:11.932252 911 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:11.932679 911 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:11.951093 911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f87df05c8204d94a3b1d8b8f37a9d0d: Generated new TSK 0
I20260926 12:43:11.952085 911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:43:11.956243 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 824
2026-09-26T12:43:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1438 ms)
[----------] 18 tests from AuthTokenIssuingTest (26137 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:12 dist-test-slave-m3pj krb5kdc[923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:12 dist-test-slave-m3pj krb5kdc[923](info): set up 2 sockets
Sep 26 12:43:12 dist-test-slave-m3pj krb5kdc[923](info): commencing operation
krb5kdc: starting...
W20260926 12:43:13.000650 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:43:13.136531 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:45313
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:45313,127.10.224.189:41181,127.10.224.188:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:13.262200 936 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:43:13.262526 936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:13.262552 936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:13.262596 936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:13.266963 936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:13.267060 936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:13.267082 936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:13.267100 936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:13.267117 936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:13.272774 936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:45313,127.10.224.189:41181,127.10.224.188:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:13.273907 936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:13.275005 936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:13.283991 942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:13.284086 941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:13.284132 936 server_base.cc:1034] running on GCE node
W20260926 12:43:13.284224 944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:13.284847 936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:13.286073 936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:13.287289 936 hybrid_clock.cc:648] HybridClock initialized: now 1790426593287259 us; error 44 us; skew 500 ppm
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:13.297847 936 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:43:13.298905 936 webserver.cc:460] Webserver started at http://127.10.224.190:35773/ using document root <none> and password file <none>
I20260926 12:43:13.299603 936 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:13.299674 936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:13.299901 936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:13.302117 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "67058b6db02940dbbba4cebbce2781ed"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "ffb0cb834c11443dfcba70f150d45316"
server_key_iv: "6a9617e9e1c3afd38e35950bf97350a0"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.302800 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "67058b6db02940dbbba4cebbce2781ed"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "ffb0cb834c11443dfcba70f150d45316"
server_key_iv: "6a9617e9e1c3afd38e35950bf97350a0"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.307067 936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:43:13.309932 951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:13.311228 936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 12:43:13.311405 936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "67058b6db02940dbbba4cebbce2781ed"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "ffb0cb834c11443dfcba70f150d45316"
server_key_iv: "6a9617e9e1c3afd38e35950bf97350a0"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.311563 936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:13.339318 936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:13.343055 936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:13.343324 936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:13.351258 936 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:45313
I20260926 12:43:13.351651 1003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:45313 every 8 connection(s)
I20260926 12:43:13.352416 936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:13.354699 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 936
I20260926 12:43:13.354887 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:13.355167 11138 external_mini_cluster.cc:1428] Setting key d59ae1a9663b6e17d6905adb7afe793c
I20260926 12:43:13.355468 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:13.360831 1004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.224.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.189@KRBTEST.COM" created.
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:43:13.377564 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.363302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:45313 (local address 127.0.0.1:57258)
0926 12:43:13.363854 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.363906 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.364792 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.365580 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.365585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.365921 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.366194 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.366207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.367877 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.367884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.369032 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.369042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.370072 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.370075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.370162 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.370781 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.370797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.372452 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.374548 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.374560 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.374576 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.374895 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.375277 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.375283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.375286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.375441 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.376110 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.376116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.376246 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.376388 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.376739 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":4,"thread_start_us":195,"threads_started":1}
I20260926 12:43:13.377590 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.363302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.10.224.190:45313)
0926 12:43:13.363927 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.363941 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.364003 (+ 62us) server_negotiation.cc:407] Connection header received
0926 12:43:13.365173 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.365178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.365366 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.365544 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.366423 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.367737 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.369175 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.369858 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.370061 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.372617 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.372623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.372633 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.372655 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.374424 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.375039 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.375047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.375052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.375168 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.375590 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.375593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.375594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.375991 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.376123 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.376493 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.376820 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":489,"spinlock_wait_cycles":4480,"thread_start_us":190,"threads_started":1}
I20260926 12:43:13.380134 1004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:13.380954 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.380688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46680)
0926 12:43:13.380822 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.380894 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 12:43:13.381165 954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
I20260926 12:43:13.381937 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.381744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46694)
0926 12:43:13.381848 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.381890 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:43:13.382174 1004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:41181: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab.
Entry for principal kudu/127.10.224.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab.
I20260926 12:43:13.388038 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.189:41181
--webserver_interface=127.10.224.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab
--principal=kudu/127.10.224.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:45313,127.10.224.189:41181,127.10.224.188:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:43:13.431600 1004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } attempt: 1
I20260926 12:43:13.432582 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.432255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46702)
0926 12:43:13.432412 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.432501 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 12:43:13.433350 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.433135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46710)
0926 12:43:13.433245 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.433297 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:43:13.433486 1004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:41181: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
I20260926 12:43:13.501657 1004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } attempt: 2
I20260926 12:43:13.502513 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.502199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46712)
0926 12:43:13.502361 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.502433 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 12:43:13.503249 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.503027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46726)
0926 12:43:13.503139 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.503191 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:43:13.503422 1004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:41181: Network error: Client connection negotiation failed: client connection to 127.10.224.189:41181: connect: Connection refused (error 111)
W20260926 12:43:13.509435 1011 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:43:13.509822 1011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:13.509897 1011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:13.509981 1011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:13.514238 1011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:13.514344 1011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:13.514379 1011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:13.514411 1011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:13.514468 1011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:13.520255 1011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:45313,127.10.224.189:41181,127.10.224.188:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab
--principal=kudu/127.10.224.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.189:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.224.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:13.521494 1011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:13.522621 1011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:13.531284 1017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:13.531288 1019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:13.531279 1016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:13.531771 1011 server_base.cc:1034] running on GCE node
I20260926 12:43:13.532299 1011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:13.533525 1011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:13.534691 1011 hybrid_clock.cc:648] HybridClock initialized: now 1790426593534658 us; error 46 us; skew 500 ppm
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:13.537647 1011 init.cc:377] Logged in from keytab as kudu/127.10.224.189@KRBTEST.COM (short username kudu)
I20260926 12:43:13.538517 1011 webserver.cc:460] Webserver started at http://127.10.224.189:33435/ using document root <none> and password file <none>
I20260926 12:43:13.539058 1011 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:13.539119 1011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:13.539299 1011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:13.541450 1011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/data/instance:
uuid: "4cb0cbe844de4f5f890514df4e02a4e3"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "b64908f579519687e010796fa13827ee"
server_key_iv: "967c82b16b99ff3d5e29a8953377f312"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.541972 1011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/wal/instance:
uuid: "4cb0cbe844de4f5f890514df4e02a4e3"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "b64908f579519687e010796fa13827ee"
server_key_iv: "967c82b16b99ff3d5e29a8953377f312"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.546025 1011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:43:13.548971 1026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:13.550784 1011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:43:13.550946 1011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/wal
uuid: "4cb0cbe844de4f5f890514df4e02a4e3"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "b64908f579519687e010796fa13827ee"
server_key_iv: "967c82b16b99ff3d5e29a8953377f312"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.551091 1011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:13.564333 1011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:13.567215 1011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:13.567440 1011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:13.574465 1011 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.189:41181
I20260926 12:43:13.574779 1078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.189:41181 every 8 connection(s)
I20260926 12:43:13.575664 1011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
I20260926 12:43:13.578462 1079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:13.582487 1079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:13.584959 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 1011
I20260926 12:43:13.585124 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-1/wal/instance
I20260926 12:43:13.585431 11138 external_mini_cluster.cc:1428] Setting key 9c6322df537bbcadca3a53458b120dc4
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.188@KRBTEST.COM; defaulting to no policy
I20260926 12:43:13.594610 1004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.188@KRBTEST.COM" created.
I20260926 12:43:13.598467 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.584772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.10.224.190:45313)
0926 12:43:13.584948 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.584953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.586108 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 12:43:13.587162 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.587167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.587244 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.587357 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.588694 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.589243 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.590460 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.590709 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.590831 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.593566 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.593570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.593572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.593601 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.595637 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.596385 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.596390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.596392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.596488 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.596965 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.596972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.596976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.597233 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.597340 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.598087 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.598236 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 12:43:13.599401 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.585285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:45313 (local address 127.0.0.1:57264)
0926 12:43:13.585891 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.585952 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.586908 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.587600 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.587609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.587983 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.588516 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.588530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.589409 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.589412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.590315 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.590325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.590882 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.590886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.591081 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.591788 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.591806 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.593359 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.595807 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.595815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.595832 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.596244 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.596599 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.596604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.596606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.596839 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.597351 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.597358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.597776 (+ 418us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.597997 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.598327 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":213,"threads_started":1}
I20260926 12:43:13.600988 1079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
I20260926 12:43:13.607074 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.595142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46736)
0926 12:43:13.595260 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.595276 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.595362 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.596191 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.596194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.596367 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.596973 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.596988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.598429 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.598433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.599202 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.599213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.600030 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.600037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.600194 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.600741 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.600760 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.602467 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.605265 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.605269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.605272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.605578 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.605893 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.605897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.605899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.605997 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.606492 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.606495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.606606 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.606725 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.606878 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 12:43:13.607128 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.595309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.10.224.189:41181)
0926 12:43:13.595696 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.595712 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.595739 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:43:13.595789 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.595795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.596057 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.596210 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.597089 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.598303 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.599349 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.599885 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.600453 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.602617 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.602631 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.602650 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.602677 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.605127 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.605687 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.605693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.605695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.605785 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.606093 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.606098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.606100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.606361 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.606522 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.606815 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.606960 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
I20260926 12:43:13.609926 1004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:13.610994 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.610626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40844)
0926 12:43:13.610775 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.610865 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 12:43:13.611977 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.611586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40850)
0926 12:43:13.611806 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.611871 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260926 12:43:13.612212 1004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:33365: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
I20260926 12:43:13.613831 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.601559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46738)
0926 12:43:13.601686 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.601701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.601780 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.602286 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.602289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.602480 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.602592 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.602598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.603809 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.603812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.604331 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.604339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.605019 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.605023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.605124 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.605641 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.605662 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.609653 (+ 3991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.611991 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.612001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.612004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.612292 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.612703 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.612706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.612707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.612787 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.613281 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.613283 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.613369 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.613558 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.613710 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 12:43:13.614074 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.601754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.10.224.189:41181)
0926 12:43:13.602052 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.602058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.602090 (+ 32us) server_negotiation.cc:407] Connection header received
0926 12:43:13.602127 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.602130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.602173 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.602278 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.602726 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.603704 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.604462 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.604871 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.605374 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.609844 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.609848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.609854 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.609884 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.611887 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.612517 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.612522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.612525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.612612 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.612934 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.612941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.612942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.613184 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.613271 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.613616 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.613822 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab.
Entry for principal kudu/127.10.224.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab.
I20260926 12:43:13.614956 1079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:13.615451 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.188:33365
--webserver_interface=127.10.224.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab
--principal=kudu/127.10.224.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:45313,127.10.224.189:41181,127.10.224.188:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:43:13.615729 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.615408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40860)
0926 12:43:13.615548 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.615654 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:43:13.615942 1027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
I20260926 12:43:13.616797 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.616592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40864)
0926 12:43:13.616698 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.616742 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:43:13.617014 1079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:33365: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
I20260926 12:43:13.643437 1004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } attempt: 1
I20260926 12:43:13.644373 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.644066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40878)
0926 12:43:13.644223 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.644295 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 12:43:13.645200 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.644958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40880)
0926 12:43:13.645083 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.645142 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 12:43:13.645342 1004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:33365: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
I20260926 12:43:13.666364 1079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } attempt: 1
I20260926 12:43:13.667412 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.667036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40886)
0926 12:43:13.667228 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.667305 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 12:43:13.668305 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.668052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40890)
0926 12:43:13.668174 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.668230 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 12:43:13.668510 1079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:33365: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
I20260926 12:43:13.720525 1004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } attempt: 2
I20260926 12:43:13.721406 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.721083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40900)
0926 12:43:13.721231 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.721302 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 12:43:13.722185 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.721983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40908)
0926 12:43:13.722089 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.722134 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:43:13.722429 1004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:33365: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
W20260926 12:43:13.735471 1087 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:43:13.735852 1087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:13.735918 1087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:13.735999 1087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:43:13.736689 1079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } attempt: 2
I20260926 12:43:13.737620 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.737242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40920)
0926 12:43:13.737459 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.737527 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 12:43:13.738401 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.738182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40930)
0926 12:43:13.738281 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.738335 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 12:43:13.738592 1079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:33365: Network error: Client connection negotiation failed: client connection to 127.10.224.188:33365: connect: Connection refused (error 111)
W20260926 12:43:13.740867 1087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:13.740922 1087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:13.740942 1087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:13.740959 1087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:13.741009 1087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:13.746723 1087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:45313,127.10.224.189:41181,127.10.224.188:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab
--principal=kudu/127.10.224.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.188:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.224.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:13.747960 1087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:13.749091 1087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:13.758051 1095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:13.758054 1092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:13.758054 1093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:13.758338 1087 server_base.cc:1034] running on GCE node
I20260926 12:43:13.758908 1087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:13.760138 1087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:13.761333 1087 hybrid_clock.cc:648] HybridClock initialized: now 1790426593761319 us; error 25 us; skew 500 ppm
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:13.764156 1087 init.cc:377] Logged in from keytab as kudu/127.10.224.188@KRBTEST.COM (short username kudu)
I20260926 12:43:13.765057 1087 webserver.cc:460] Webserver started at http://127.10.224.188:33779/ using document root <none> and password file <none>
I20260926 12:43:13.765632 1087 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:13.765726 1087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:13.765969 1087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:13.768131 1087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/data/instance:
uuid: "5149fd52d72f4c8c8477a1c672ea1ba1"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "8d6618b802fe71f098ae506f0ec34c2c"
server_key_iv: "af7ba049f9ad1097d41f5c2adda2856d"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.768698 1087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/wal/instance:
uuid: "5149fd52d72f4c8c8477a1c672ea1ba1"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "8d6618b802fe71f098ae506f0ec34c2c"
server_key_iv: "af7ba049f9ad1097d41f5c2adda2856d"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.772593 1087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:43:13.775203 1102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:13.776594 1087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:43:13.776752 1087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/wal
uuid: "5149fd52d72f4c8c8477a1c672ea1ba1"
format_stamp: "Formatted at 2026-09-26 12:43:13 on dist-test-slave-m3pj"
server_key: "8d6618b802fe71f098ae506f0ec34c2c"
server_key_iv: "af7ba049f9ad1097d41f5c2adda2856d"
server_key_version: "encryptionkey@0"
I20260926 12:43:13.776885 1087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:13.791900 1087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:13.795400 1087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:13.795687 1087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:13.803591 1087 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.188:33365
I20260926 12:43:13.803925 1154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.188:33365 every 8 connection(s)
I20260926 12:43:13.804721 1087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
I20260926 12:43:13.807802 1155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:13.811734 1155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:13.813122 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 1087
I20260926 12:43:13.813301 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790426298078858-11138-0/minicluster-data/master-2/wal/instance
I20260926 12:43:13.813591 11138 external_mini_cluster.cc:1428] Setting key a74c329228d45bdab2847a4524e96606
I20260926 12:43:13.821617 1004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } attempt: 3
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:43:13.826860 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.813895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.10.224.190:45313)
0926 12:43:13.814081 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.814088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.815085 (+ 997us) server_negotiation.cc:407] Connection header received
0926 12:43:13.816165 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.816170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.816258 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.816352 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.817287 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.817813 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.818905 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.819254 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.819363 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.822602 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.822607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.822610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.822638 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.824528 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.825097 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.825101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.825103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.825220 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.825588 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.825591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.825593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.825842 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.825939 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.826542 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.826689 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63,"spinlock_wait_cycles":1920}
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:43:13.828007 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.814346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.10.224.188:33365)
0926 12:43:13.814825 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.814844 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.814884 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:43:13.815869 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.815893 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.816136 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.816286 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.817102 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.818129 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.819027 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.819655 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.819735 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.822934 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.822941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.822962 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.822983 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.824743 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.825284 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.825290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.825292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.825424 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.825741 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.825745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.825747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.826294 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.826523 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.827581 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.827723 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":332,"thread_start_us":149,"threads_started":1}
I20260926 12:43:13.828265 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.814345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:45313 (local address 127.0.0.1:57274)
0926 12:43:13.814955 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.815000 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.815869 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.816425 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.816431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.816842 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.817147 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.817162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.817923 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.817927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.818779 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.818788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.819406 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.819409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.819640 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.820279 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.820305 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.822409 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.824652 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.824661 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.824674 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.824990 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.825330 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.825338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.825340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.825458 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.825948 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.825955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.826295 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.826494 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.826851 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":31,"spinlock_wait_cycles":61056,"thread_start_us":179,"threads_started":1}
I20260926 12:43:13.829604 1155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:13.829934 1079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } attempt: 3
I20260926 12:43:13.832301 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.822137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40940)
0926 12:43:13.822368 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.822384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.822467 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.823568 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.823573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.823760 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.823865 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.823871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.824980 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.824984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.825745 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.825755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.826361 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.826364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.826456 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.826933 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.826947 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.828378 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.830693 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.830697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.830699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.830954 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.831244 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.831247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.831249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.831335 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.831822 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.831828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.831931 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.832009 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.832137 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"spinlock_wait_cycles":1408}
I20260926 12:43:13.832878 1160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.822184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.10.224.188:33365)
0926 12:43:13.822741 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.822751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.822766 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:43:13.822801 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.822804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.822856 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.823487 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.824113 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.824871 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.825879 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.826234 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.826345 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.828529 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.828533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.828538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.828563 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.830483 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.831044 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.831049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.831051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.831141 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.831441 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.831443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.831445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.831699 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.832458 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.832545 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.832664 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":486,"thread_start_us":145,"threads_started":1}
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
I20260926 12:43:13.836715 1004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed: Bootstrap starting.
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:43:13.840263 1004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:13.841403 1004 log.cc:824] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:13.841851 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.830904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.10.224.190:45313)
0926 12:43:13.831011 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.831015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.831028 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:43:13.831062 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.831064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.831103 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.831179 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.832037 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.832760 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.834185 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.834498 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.834568 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.837466 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.837470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.837473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.837500 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.839263 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.839862 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.839869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.839873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.839989 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.840562 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.840567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.840570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.840831 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.840967 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.841564 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.841696 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:43:13.841908 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.830366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.10.224.189:41181)
0926 12:43:13.830483 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.830488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.830640 (+ 152us) server_negotiation.cc:407] Connection header received
0926 12:43:13.831204 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.831209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.831258 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.831308 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.831870 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.832610 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.833723 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.834027 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.834089 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.837392 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.837396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.837399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.837429 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.839156 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.840043 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.840048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.840053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.840152 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.840781 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.840785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.840787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.841087 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.841194 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.841543 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.841703 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:43:13.842429 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.830131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:41181 (local address 127.0.0.1:46748)
0926 12:43:13.830583 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.830597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.831085 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.831442 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.831445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.831640 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.831752 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.831761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.832846 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.832851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.833561 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.833569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.834210 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.834212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.834280 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.834768 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.834789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.837226 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.839283 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.839287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.839289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.839904 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.840279 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.840283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.840286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.840656 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.841205 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.841211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.841316 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.842052 (+ 736us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.842216 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370}
I20260926 12:43:13.843359 1155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:13.843688 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.830348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40950)
0926 12:43:13.830600 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.830614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.830689 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.830986 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.830990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.831186 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.831290 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.831297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.832621 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.832624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.833425 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.833435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.836016 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.836023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.836116 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.836564 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.836578 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.838514 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.841747 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.841750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.841752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.841982 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.842398 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.842401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.842403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.842498 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.843131 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.843134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.843241 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.843363 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.843509 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 12:43:13.844339 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.830342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.10.224.188:33365)
0926 12:43:13.830646 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.830650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.830661 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:43:13.830801 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.830804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.830852 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.831398 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.831491 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.832289 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.833675 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.835854 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.837739 (+ 1885us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.839318 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.839322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.839328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.839356 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.841631 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.842194 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.842197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.842199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.842273 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.842626 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.842784 (+ 158us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.842787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.843016 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.843948 (+ 932us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.844043 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.844170 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260926 12:43:13.844616 1004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed: No bootstrap required, opened a new log
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:43:13.849282 1004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } }
I20260926 12:43:13.849591 1004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67058b6db02940dbbba4cebbce2781ed, State: Initialized, Role: FOLLOWER
I20260926 12:43:13.850019 1004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } }
Sep 26 12:43:13 dist-test-slave-m3pj krb5kdc[923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
I20260926 12:43:13.852777 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.843980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:33365 (local address 127.0.0.1:40952)
0926 12:43:13.844111 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:13.844126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:13.844197 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:13.845114 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:13.845118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:13.845257 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:13.845321 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.845325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.846080 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.846086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.846604 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:13.846611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.847077 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.847080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:13.847179 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.847728 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:13.847743 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:13.849069 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:13.851113 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.851116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.851119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.851367 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.851721 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:13.851723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:13.851725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:13.851808 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:13.852247 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:13.852250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:13.852346 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 12:43:13.852458 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 12:43:13.852598 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":4}
I20260926 12:43:13.852916 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.844554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.10.224.188:33365)
0926 12:43:13.844835 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.844839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.844850 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:43:13.844881 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.844887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.844932 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.845014 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.845429 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.845966 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.846721 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.846962 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.847067 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.849209 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.849213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.849215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.849237 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.850983 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.851538 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.851543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.851544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.851636 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.851898 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.851903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.851905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.852156 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.852261 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.852549 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.852681 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":222}
I20260926 12:43:13.854100 1079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3: Bootstrap starting.
I20260926 12:43:13.855645 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.844658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.10.224.189:41181)
0926 12:43:13.844764 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.844768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.844937 (+ 169us) server_negotiation.cc:407] Connection header received
0926 12:43:13.845134 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.845137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.845181 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.845230 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.846135 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.846863 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.847998 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.848359 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.848441 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.851720 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.851724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.851727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.851755 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.853530 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.854050 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.854053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.854055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.854145 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.854473 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.854476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.854478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.854729 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.854923 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.855303 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.855427 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:43:13.856272 1155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1: Bootstrap starting.
I20260926 12:43:13.857201 1079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:13.858112 1079 log.cc:824] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:13.851496 1004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:13.858659 1155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:13.859503 1155 log.cc:824] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:13.859709 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:13.861020 978 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "67058b6db02940dbbba4cebbce2781ed"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57280
I20260926 12:43:13.861260 978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 12:43:13.861370 978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:13.861775 1155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1: No bootstrap required, opened a new log
I20260926 12:43:13.862242 1079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3: No bootstrap required, opened a new log
I20260926 12:43:13.862766 978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } }
I20260926 12:43:13.863811 978 leader_election.cc:288] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [CANDIDATE]: Term 1 election: Requested vote from peers 4cb0cbe844de4f5f890514df4e02a4e3 (127.10.224.189:41181), 5149fd52d72f4c8c8477a1c672ea1ba1 (127.10.224.188:33365)
I20260926 12:43:13.865649 1155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } }
I20260926 12:43:13.865922 1155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5149fd52d72f4c8c8477a1c672ea1ba1, State: Initialized, Role: FOLLOWER
I20260926 12:43:13.866299 1155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } }
I20260926 12:43:13.866456 1079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } }
I20260926 12:43:13.866822 1079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb0cbe844de4f5f890514df4e02a4e3, State: Initialized, Role: FOLLOWER
I20260926 12:43:13.867882 1155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:13.868848 1129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67058b6db02940dbbba4cebbce2781ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1"
I20260926 12:43:13.869125 1129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:13.869495 1079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } }
I20260926 12:43:13.870599 1129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67058b6db02940dbbba4cebbce2781ed in term 1.
I20260926 12:43:13.870800 1181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.870949 1181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:13.871134 952 leader_election.cc:302] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5149fd52d72f4c8c8477a1c672ea1ba1, 67058b6db02940dbbba4cebbce2781ed; no voters:
I20260926 12:43:13.870847 1162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.872473 1197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:13.873692 1162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: This master's current role is: FOLLOWER
W20260926 12:43:13.874033 1196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:43:13.874110 1196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:43:13.874046 1197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [term 1 LEADER]: Becoming Leader. State: Replica: 67058b6db02940dbbba4cebbce2781ed, State: Running, Role: LEADER
I20260926 12:43:13.874516 1079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:13.874464 1197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } }
I20260926 12:43:13.874564 1053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67058b6db02940dbbba4cebbce2781ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4cb0cbe844de4f5f890514df4e02a4e3"
I20260926 12:43:13.874814 1053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:13.876274 1053 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67058b6db02940dbbba4cebbce2781ed in term 1.
I20260926 12:43:13.873044 936 master_runner.cc:186] Error getting master registration for 127.10.224.189:41181: OK, instance_id { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" instance_seqno: 1790426593572432 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:43:13.876636 936 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:43:13.877728 1190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.879596 1190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 12:43:13.883111 1210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:43:13.883204 1210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:43:13.883877 1162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67058b6db02940dbbba4cebbce2781ed. Latest consensus state: current_term: 1 leader_uuid: "67058b6db02940dbbba4cebbce2781ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.884060 1162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: This master's current role is: LEADER
W20260926 12:43:13.873600 1175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:43:13.884343 1175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:43:13.887856 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.866959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.10.224.188:33365)
0926 12:43:13.867309 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.867313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.878118 (+ 10805us) server_negotiation.cc:407] Connection header received
0926 12:43:13.878319 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.878323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.878377 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.878448 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.879401 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.880107 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.881454 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.881784 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.881884 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.883896 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.883901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.883907 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.883939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.885843 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.886372 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.886376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.886393 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.886486 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.886793 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.886797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.886799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.887037 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.887142 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.887562 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.887704 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260926 12:43:13.892854 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.866350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.10.224.189:41181)
0926 12:43:13.866488 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.866493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.867641 (+ 1148us) server_negotiation.cc:407] Connection header received
0926 12:43:13.867836 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.867839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.867889 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.867957 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.869875 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.870687 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.877276 (+ 6589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.877590 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.877677 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.879676 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.879680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.879683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.879710 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.881629 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.891271 (+ 9642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.891275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.891277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.891403 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.891795 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.891799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.891801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.892044 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.892169 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.892543 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.892683 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:43:13.893621 1213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:13.894737 1213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:43:13.900851 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:13.869350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.10.224.190:45313)
0926 12:43:13.869487 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:13.869493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:13.880061 (+ 10568us) server_negotiation.cc:407] Connection header received
0926 12:43:13.880428 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:13.880434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:13.880490 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:13.880561 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:13.881662 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.882385 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.893339 (+ 10954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:13.893702 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:13.893813 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:13.896954 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:13.896958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:13.896961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:13.896993 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:13.898871 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.899381 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.899385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.899387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.899470 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:13.899839 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:13.899842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:13.899844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:13.900074 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:13.900187 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:13.900560 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 12:43:13.900688 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:43:13.911850 1129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [term 1 FOLLOWER]: Refusing update from remote peer 67058b6db02940dbbba4cebbce2781ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:43:13.913141 1199 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:43:13.917379 1181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67058b6db02940dbbba4cebbce2781ed. Latest consensus state: current_term: 1 leader_uuid: "67058b6db02940dbbba4cebbce2781ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.917517 1181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:13.918632 1053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [term 1 FOLLOWER]: Refusing update from remote peer 67058b6db02940dbbba4cebbce2781ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:43:13.921725 1199 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:43:13.925933 1190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67058b6db02940dbbba4cebbce2781ed. Latest consensus state: current_term: 1 leader_uuid: "67058b6db02940dbbba4cebbce2781ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.926082 1190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:13.934973 1199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67058b6db02940dbbba4cebbce2781ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.935148 1199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:13.936388 1190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67058b6db02940dbbba4cebbce2781ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.936527 1190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:13.936952 1214 mvcc.cc:205] Tried to move back new op lower bound from 7333587328653504512 to 7333587328533065728. Current Snapshot: MvccSnapshot[applied={T|T < 7333587328653504512}]
I20260926 12:43:13.938279 1217 mvcc.cc:205] Tried to move back new op lower bound from 7333587328653504512 to 7333587328533065728. Current Snapshot: MvccSnapshot[applied={T|T < 7333587328653504512}]
I20260926 12:43:13.940721 1162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67058b6db02940dbbba4cebbce2781ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.940884 1162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:13.944691 1181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67058b6db02940dbbba4cebbce2781ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67058b6db02940dbbba4cebbce2781ed" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 45313 } } peers { permanent_uuid: "4cb0cbe844de4f5f890514df4e02a4e3" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 41181 } } peers { permanent_uuid: "5149fd52d72f4c8c8477a1c672ea1ba1" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 33365 } } }
I20260926 12:43:13.944837 1181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:13.945606 1215 mvcc.cc:205] Tried to move back new op lower bound from 7333587328653504512 to 7333587328533065728. Current Snapshot: MvccSnapshot[applied={T|T < 7333587328653504512}]
I20260926 12:43:13.948282 1213 catalog_manager.cc:1347] Generated new cluster ID: c676a5a19f754d3990de34b086b9974d
I20260926 12:43:13.948369 1213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:13.971648 1213 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:13.973035 1213 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:13.993053 1213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67058b6db02940dbbba4cebbce2781ed: Generated new TSK 0
I20260926 12:43:13.994529 1213 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:43:14.036321 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.033826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46772 (local address 127.10.224.189:41181)
0926 12:43:14.033997 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.034003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.035199 (+ 1196us) server_negotiation.cc:407] Connection header received
0926 12:43:14.035394 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.035398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.035448 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.035501 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.036182 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46772: BlockingRecv error: recv got EOF from 127.0.0.1:46772 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 12:43:14.037735 1158 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.034532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.10.224.188:33365)
0926 12:43:14.034649 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.034654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.036829 (+ 2175us) server_negotiation.cc:407] Connection header received
0926 12:43:14.037029 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.037032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.037081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.037133 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.037613 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40968: BlockingRecv error: recv got EOF from 127.0.0.1:40968 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 12:43:14.039929 1008 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.038598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.10.224.190:45313)
0926 12:43:14.038734 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.038739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.038947 (+ 208us) server_negotiation.cc:407] Connection header received
0926 12:43:14.039113 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.039117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.039164 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.039215 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.039797 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57292: BlockingRecv error: recv got EOF from 127.0.0.1:57292 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 12:43:14.044602 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.043004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.10.224.189:41181)
0926 12:43:14.043114 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.043119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.043613 (+ 494us) server_negotiation.cc:407] Connection header received
0926 12:43:14.043834 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.043837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.043927 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.043984 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.044498 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46784: BlockingRecv error: recv got EOF from 127.0.0.1:46784 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 12:43:14.050040 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.048339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.10.224.189:41181)
0926 12:43:14.048535 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.048539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.049063 (+ 524us) server_negotiation.cc:407] Connection header received
0926 12:43:14.049245 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.049248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.049302 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.049357 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.049919 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46796: BlockingRecv error: recv got EOF from 127.0.0.1:46796 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260926 12:43:14.056262 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.054413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.10.224.189:41181)
0926 12:43:14.054542 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.054547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.055126 (+ 579us) server_negotiation.cc:407] Connection header received
0926 12:43:14.055333 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.055336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.055397 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.055452 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.056141 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46798: BlockingRecv error: recv got EOF from 127.0.0.1:46798 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 12:43:14.064482 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.062627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.10.224.189:41181)
0926 12:43:14.062777 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.062782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.063374 (+ 592us) server_negotiation.cc:407] Connection header received
0926 12:43:14.063630 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.063635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.063696 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.063753 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.064350 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46810: BlockingRecv error: recv got EOF from 127.0.0.1:46810 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 12:43:14.077561 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.075498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.10.224.189:41181)
0926 12:43:14.075679 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.075685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.076390 (+ 705us) server_negotiation.cc:407] Connection header received
0926 12:43:14.076636 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.076640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.076701 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.076762 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.077432 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46822: BlockingRecv error: recv got EOF from 127.0.0.1:46822 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260926 12:43:14.098575 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.096364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.10.224.189:41181)
0926 12:43:14.096492 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.096499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.097128 (+ 629us) server_negotiation.cc:407] Connection header received
0926 12:43:14.097506 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.097516 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.097613 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.097718 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.098433 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46824: BlockingRecv error: recv got EOF from 127.0.0.1:46824 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 12:43:14.135123 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.133909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.10.224.189:41181)
0926 12:43:14.134081 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.134088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.134270 (+ 182us) server_negotiation.cc:407] Connection header received
0926 12:43:14.134378 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.134381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.134431 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.134505 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.134977 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46838: BlockingRecv error: recv got EOF from 127.0.0.1:46838 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 12:43:14.202657 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.201330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.10.224.189:41181)
0926 12:43:14.201547 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.201552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.201670 (+ 118us) server_negotiation.cc:407] Connection header received
0926 12:43:14.201881 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.201887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.201942 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.202025 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.202538 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46854: BlockingRecv error: recv got EOF from 127.0.0.1:46854 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 12:43:14.334059 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.333043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.10.224.189:41181)
0926 12:43:14.333194 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.333198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.333218 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:43:14.333279 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.333283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.333336 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.333429 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.333938 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53310: BlockingRecv error: recv got EOF from 127.0.0.1:53310 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 12:43:14.593726 1085 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:14.592460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.10.224.189:41181)
0926 12:43:14.592628 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:14.592631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:14.592694 (+ 63us) server_negotiation.cc:407] Connection header received
0926 12:43:14.592828 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:14.592830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:14.592883 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:14.592961 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:14.593585 (+ 624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53314: BlockingRecv error: recv got EOF from 127.0.0.1:53314 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 12:43:14.875633 1196 catalog_manager.cc:1259] Loaded cluster ID: c676a5a19f754d3990de34b086b9974d
I20260926 12:43:14.875716 1196 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1: loading cluster ID for follower catalog manager: success
I20260926 12:43:14.877619 1196 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1: acquiring CA information for follower catalog manager: success
I20260926 12:43:14.878841 1196 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5149fd52d72f4c8c8477a1c672ea1ba1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:43:14.884446 1210 catalog_manager.cc:1259] Loaded cluster ID: c676a5a19f754d3990de34b086b9974d
I20260926 12:43:14.884519 1210 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3: loading cluster ID for follower catalog manager: success
I20260926 12:43:14.886251 1210 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3: acquiring CA information for follower catalog manager: success
I20260926 12:43:14.887578 1210 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4cb0cbe844de4f5f890514df4e02a4e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:43:15.112296 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:15.108003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.10.224.189:41181)
0926 12:43:15.108310 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:15.108314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:15.108331 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:43:15.108410 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:15.108412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:15.108555 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:15.108659 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:15.108967 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.109544 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.110534 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.110770 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.110889 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:15.111239 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:15.111875 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:15.111986 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:15.112090 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 12:43:15.112157 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":83,"threads_started":1}
I20260926 12:43:15.120239 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:15.116144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.10.224.188:33365)
0926 12:43:15.116424 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:15.116429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:15.116489 (+ 60us) server_negotiation.cc:407] Connection header received
0926 12:43:15.116586 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:15.116589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:15.116709 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:15.116835 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:15.117093 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.117642 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.118528 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.118764 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.118876 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:15.119229 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:15.119785 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:15.119922 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:15.119984 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 12:43:15.120051 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260926 12:43:15.127933 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:15.123703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.10.224.189:41181)
0926 12:43:15.123837 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:15.123841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:15.124242 (+ 401us) server_negotiation.cc:407] Connection header received
0926 12:43:15.124496 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:15.124502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:15.124587 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:15.124658 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:15.125066 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.125825 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.126671 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.126919 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.127043 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:15.127384 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:15.127474 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:15.127580 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:15.127695 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 12:43:15.127762 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:43:15.128767 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:15.124109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.10.224.188:33365)
0926 12:43:15.124242 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:15.124246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:15.124259 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:43:15.124404 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:15.124410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:15.124480 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:15.124554 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:15.124819 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.125638 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.127310 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.127655 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.127746 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:15.128304 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:15.128388 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:15.128482 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:15.128562 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 12:43:15.128618 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:43:15.129130 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:15.123686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.10.224.190:45313)
0926 12:43:15.123984 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:15.123988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:15.124259 (+ 271us) server_negotiation.cc:407] Connection header received
0926 12:43:15.124569 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:15.124572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:15.124732 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:15.124866 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:15.125146 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.125665 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.127325 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:15.127722 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:15.127813 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:15.128282 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:15.128654 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:15.128745 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:15.128921 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 12:43:15.128983 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
I20260926 12:43:15.132622 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 936
I20260926 12:43:15.139403 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 1011
I20260926 12:43:15.145679 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 1087
2026-09-26T12:43:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3192 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:43:15 dist-test-slave-m3pj krb5kdc[1303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:43:15 dist-test-slave-m3pj krb5kdc[1303](info): set up 2 sockets
Sep 26 12:43:15 dist-test-slave-m3pj krb5kdc[1303](info): commencing operation
krb5kdc: starting...
W20260926 12:43:16.204792 11138 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/test-user.keytab.
Sep 26 12:43:16 dist-test-slave-m3pj krb5kdc[1303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:43:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.224.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
Entry for principal kudu/127.10.224.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab.
I20260926 12:43:16.338510 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34877
--webserver_interface=127.10.224.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:34877,127.10.224.189:39819,127.10.224.188:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:43:16.457244 1316 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:43:16.457651 1316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:16.457711 1316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:16.457798 1316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:16.462337 1316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:16.462486 1316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:16.462512 1316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:16.462530 1316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:16.462548 1316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:16.468390 1316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:34877,127.10.224.189:39819,127.10.224.188:45239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.190.keytab
--principal=kudu/127.10.224.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.190:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.224.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:16.469686 1316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:16.470862 1316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:16.479884 1324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:16.479910 1322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:16.479981 1316 server_base.cc:1034] running on GCE node
W20260926 12:43:16.479913 1321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:16.480736 1316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:16.481856 1316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:16.483094 1316 hybrid_clock.cc:648] HybridClock initialized: now 1790426596483047 us; error 41 us; skew 500 ppm
Sep 26 12:43:16 dist-test-slave-m3pj krb5kdc[1303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:16.486414 1316 init.cc:377] Logged in from keytab as kudu/127.10.224.190@KRBTEST.COM (short username kudu)
I20260926 12:43:16.487385 1316 webserver.cc:460] Webserver started at http://127.10.224.190:42363/ using document root <none> and password file <none>
I20260926 12:43:16.488019 1316 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:16.488114 1316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:16.488351 1316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:16.490432 1316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/data/instance:
uuid: "9f8360ad2e9b43d0bf8fd195031215d7"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "f8e9c4c57af8c9510a3d14139e745218"
server_key_iv: "3fde7c5afcec56686cce4c50bd28cc65"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.491025 1316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/wal/instance:
uuid: "9f8360ad2e9b43d0bf8fd195031215d7"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "f8e9c4c57af8c9510a3d14139e745218"
server_key_iv: "3fde7c5afcec56686cce4c50bd28cc65"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.494801 1316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:43:16.497524 1331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:16.498816 1316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:43:16.498968 1316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/wal
uuid: "9f8360ad2e9b43d0bf8fd195031215d7"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "f8e9c4c57af8c9510a3d14139e745218"
server_key_iv: "3fde7c5afcec56686cce4c50bd28cc65"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.499109 1316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:16.512578 1316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:16.516328 1316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:16.516551 1316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:16.523989 1316 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.190:34877
I20260926 12:43:16.524340 1383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.190:34877 every 8 connection(s)
I20260926 12:43:16.525089 1316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/data/info.pb
I20260926 12:43:16.525913 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 1316
I20260926 12:43:16.526054 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-0/wal/instance
I20260926 12:43:16.526327 11138 external_mini_cluster.cc:1428] Setting key d2c3eeef50d2e37b20173e39b45e7832
I20260926 12:43:16.528278 1384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:16.534061 1384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.224.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.189@KRBTEST.COM" created.
Sep 26 12:43:16 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:43:16.551921 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.536938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34877 (local address 127.0.0.1:48696)
0926 12:43:16.537510 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.537562 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:16.538561 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:16.539303 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:16.539312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:16.539846 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:16.540192 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:16.540208 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.542217 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.542220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:16.543067 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:16.543077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.544171 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.544174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:16.544326 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:16.544942 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:16.544958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:16.546447 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:16.548741 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:16.548749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:16.548758 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:16.548984 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:16.549424 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:16.549430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:16.549435 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:16.549601 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:16.550357 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:16.550362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:16.550494 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 12:43:16.550651 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 12:43:16.551026 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":167,"spinlock_wait_cycles":44928,"thread_start_us":178,"threads_started":1}
I20260926 12:43:16.551918 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.536938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.10.224.190:34877)
0926 12:43:16.537388 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:16.537427 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:16.537675 (+ 248us) server_negotiation.cc:407] Connection header received
0926 12:43:16.538883 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:16.538888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:16.539089 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:16.539245 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:16.540377 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.542067 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.543202 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.544004 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.544223 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:16.546630 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:16.546641 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:16.546659 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:16.546695 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:16.548634 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:16.549114 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:16.549119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:16.549121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:16.549319 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:16.549706 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:16.549708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:16.549710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:16.550249 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:16.550369 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:16.550770 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 12:43:16.550945 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":307,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab.
Entry for principal kudu/127.10.224.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab.
I20260926 12:43:16.554738 1384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:16.554796 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.189:39819
--webserver_interface=127.10.224.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab
--principal=kudu/127.10.224.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:34877,127.10.224.189:39819,127.10.224.188:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:43:16.555677 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.555299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53432)
0926 12:43:16.555478 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.555609 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 12:43:16.555955 1334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
I20260926 12:43:16.556996 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.556710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53444)
0926 12:43:16.556839 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.556916 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:43:16.557178 1384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:39819: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
I20260926 12:43:16.606513 1384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } attempt: 1
I20260926 12:43:16.607596 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.607151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53446)
0926 12:43:16.607352 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.607450 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 12:43:16.608453 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.608189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53450)
0926 12:43:16.608307 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.608367 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 12:43:16.608616 1384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:39819: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
I20260926 12:43:16.676846 1384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } attempt: 2
I20260926 12:43:16.677856 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.677448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53462)
0926 12:43:16.677660 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.677747 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 12:43:16.678850 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.678538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53470)
0926 12:43:16.678699 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.678775 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 12:43:16.679028 1384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.189:39819: Network error: Client connection negotiation failed: client connection to 127.10.224.189:39819: connect: Connection refused (error 111)
W20260926 12:43:16.682192 1390 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:43:16.682574 1390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:16.682628 1390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:16.682747 1390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:43:16.687628 1390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:16.687702 1390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:16.687724 1390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:16.687772 1390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:16.687809 1390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:16.694262 1390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:34877,127.10.224.189:39819,127.10.224.188:45239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.189.keytab
--principal=kudu/127.10.224.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.189:39819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.224.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:16.695623 1390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:16.696792 1390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:16.706899 1396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:16.706857 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:16.706830 1397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:16.707319 1390 server_base.cc:1034] running on GCE node
I20260926 12:43:16.708067 1390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:16.709421 1390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:16.710654 1390 hybrid_clock.cc:648] HybridClock initialized: now 1790426596710635 us; error 34 us; skew 500 ppm
Sep 26 12:43:16 dist-test-slave-m3pj krb5kdc[1303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:16.713944 1390 init.cc:377] Logged in from keytab as kudu/127.10.224.189@KRBTEST.COM (short username kudu)
I20260926 12:43:16.715044 1390 webserver.cc:460] Webserver started at http://127.10.224.189:33905/ using document root <none> and password file <none>
I20260926 12:43:16.715732 1390 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:16.715829 1390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:16.716073 1390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:16.718206 1390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/data/instance:
uuid: "f9ecc90e38ec41aaa9466516a187a047"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "eea6e22daa7dc201a2aa96a3ba920db0"
server_key_iv: "e7cde147970889b8ab10048f3a98ce32"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.718793 1390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/wal/instance:
uuid: "f9ecc90e38ec41aaa9466516a187a047"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "eea6e22daa7dc201a2aa96a3ba920db0"
server_key_iv: "e7cde147970889b8ab10048f3a98ce32"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.722615 1390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 12:43:16.725234 1406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:16.726490 1390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:43:16.726642 1390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/wal
uuid: "f9ecc90e38ec41aaa9466516a187a047"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "eea6e22daa7dc201a2aa96a3ba920db0"
server_key_iv: "e7cde147970889b8ab10048f3a98ce32"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.726763 1390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:16.754904 1390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:16.758523 1390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:16.758738 1390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:16.765961 1390 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.189:39819
I20260926 12:43:16.766299 1458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.189:39819 every 8 connection(s)
I20260926 12:43:16.766980 1390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/data/info.pb
I20260926 12:43:16.769923 1459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:16.770247 1384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } attempt: 3
I20260926 12:43:16.772881 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 1390
I20260926 12:43:16.773031 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-1/wal/instance
I20260926 12:43:16.773350 11138 external_mini_cluster.cc:1428] Setting key c48cc8078057e82b8880bc8990b8279a
I20260926 12:43:16.775707 1459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:43:16 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.224.188@KRBTEST.COM; defaulting to no policy
I20260926 12:43:16.785979 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.770843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53478)
0926 12:43:16.771004 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.771024 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:16.771127 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:16.773926 (+ 2799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:16.773929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:16.774144 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:16.775244 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:16.775253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.777067 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.777075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:16.778157 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:16.778176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.779388 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.779392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:16.779508 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:16.780181 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:16.780195 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:16.781500 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:16.784105 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:16.784108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:16.784109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:16.784293 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:16.784736 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:16.784739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:16.784741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:16.784827 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:16.785553 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:16.785555 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:16.785637 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 12:43:16.785717 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 12:43:16.785833 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.224.188@KRBTEST.COM" created.
Sep 26 12:43:16 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:43:16.787711 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.771591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.10.224.189:39819)
0926 12:43:16.772064 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:16.772081 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:16.772118 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:43:16.773046 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:16.773075 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:16.773562 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:16.773986 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:16.775437 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.776909 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.778345 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.779256 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.779489 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:16.781675 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:16.781680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:16.781689 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:16.781715 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:16.783990 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:16.784415 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:16.784483 (+ 68us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:16.784487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:16.784607 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:16.784966 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:16.784969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:16.784970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:16.785392 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:16.785593 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:16.785877 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 12:43:16.786522 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":136,"threads_started":1}
I20260926 12:43:16.790318 1384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:16.791296 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.790980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47344)
0926 12:43:16.791138 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.791202 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 12:43:16.792141 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.777674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.10.224.190:34877)
0926 12:43:16.777814 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:16.777819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:16.778181 (+ 362us) server_negotiation.cc:407] Connection header received
0926 12:43:16.778367 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:16.778373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:16.778429 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:16.778473 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:16.779223 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.780636 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.782187 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.782469 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.782630 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:16.787567 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:16.787574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:16.787580 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:16.787613 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:16.789800 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:16.790397 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:16.790401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:16.790403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:16.790506 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:16.790845 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:16.790849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:16.790851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:16.791107 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:16.791209 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:16.791808 (+ 599us) server_negotiation.cc:299] Negotiation successful
0926 12:43:16.791991 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:43:16.792341 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.777636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34877 (local address 127.0.0.1:48708)
0926 12:43:16.778057 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.778106 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:16.778263 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:16.778613 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:16.778617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:16.778956 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:16.779100 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:16.779106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.780788 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.780791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:16.782000 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:16.782012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.782631 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.782635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:16.782957 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:16.783436 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:16.783448 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:16.787376 (+ 3928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:16.789912 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:16.789919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:16.789925 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:16.790281 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:16.790636 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:16.790639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:16.790642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:16.790743 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:16.791234 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:16.791244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:16.791477 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 12:43:16.791729 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 12:43:16.791970 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":3,"thread_start_us":150,"threads_started":1}
I20260926 12:43:16.792529 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.792002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47346)
0926 12:43:16.792197 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.792453 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260926 12:43:16.792668 1384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:45239: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
I20260926 12:43:16.793856 1459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:43:16 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
I20260926 12:43:16.803623 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.794394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53486)
0926 12:43:16.794614 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.794628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:16.794730 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:16.795003 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:16.795007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:16.795205 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:16.795354 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:16.795363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.796289 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.796293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:16.797126 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:16.797136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.797670 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.797681 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:16.797761 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:16.798263 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:16.798275 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:16.799770 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:16.801826 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:16.801829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:16.801830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:16.802058 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:16.802445 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:16.802448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:16.802449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:16.802535 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:16.802974 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:16.802976 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:16.803057 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 12:43:16.803151 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:43:16.803290 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 12:43:16.803676 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.794499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.10.224.189:39819)
0926 12:43:16.794580 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:16.794585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:16.794696 (+ 111us) server_negotiation.cc:407] Connection header received
0926 12:43:16.794824 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:16.794826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:16.794865 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:16.794955 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:16.795589 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.796167 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.797241 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:16.797520 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:16.797608 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:16.799931 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:16.799938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:16.799945 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:16.799979 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:16.801716 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:16.802207 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:16.802211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:16.802212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:16.802305 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:16.802609 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:16.802612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:16.802613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:16.802869 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:16.802977 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:16.803211 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 12:43:16.803362 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 12:43:16.804494 1459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:16.805361 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.804990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47352)
0926 12:43:16.805157 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.805270 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 12:43:16.805706 1408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
I20260926 12:43:16.806754 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.806537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47364)
0926 12:43:16.806646 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.806691 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:43:16.806933 1459 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:45239: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.224.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab.
Entry for principal kudu/127.10.224.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab.
I20260926 12:43:16.808024 11138 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
/tmp/dist-test-taskrbzlsq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.188:45239
--webserver_interface=127.10.224.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab
--principal=kudu/127.10.224.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.224.148:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.224.190:34877,127.10.224.189:39819,127.10.224.188:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:43:16.823837 1384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } attempt: 1
I20260926 12:43:16.824782 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.824448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47374)
0926 12:43:16.824627 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.824697 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 12:43:16.825634 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.825358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47390)
0926 12:43:16.825529 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.825575 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 12:43:16.825764 1384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:45239: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
I20260926 12:43:16.856232 1459 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } attempt: 1
I20260926 12:43:16.857232 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.856909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47392)
0926 12:43:16.857078 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.857148 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 12:43:16.858089 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.857822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47402)
0926 12:43:16.857975 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.858024 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:43:16.858234 1459 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:45239: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
I20260926 12:43:16.900987 1384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } attempt: 2
I20260926 12:43:16.902107 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.901626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47414)
0926 12:43:16.901839 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.901986 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 12:43:16.903091 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.902827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47430)
0926 12:43:16.902955 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.903029 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:43:16.903225 1384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:45239: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
W20260926 12:43:16.924578 1466 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:43:16.924939 1466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:43:16.925010 1466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:43:16.925092 1466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:43:16.926424 1459 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } attempt: 2
I20260926 12:43:16.927306 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.927003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47432)
0926 12:43:16.927161 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.927232 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 12:43:16.928011 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.927799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47440)
0926 12:43:16.927902 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:16.927955 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:43:16.928131 1459 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.224.188:45239: Network error: Client connection negotiation failed: client connection to 127.10.224.188:45239: connect: Connection refused (error 111)
W20260926 12:43:16.929625 1466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:43:16.929719 1466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:43:16.929754 1466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:43:16.929772 1466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:43:16.929797 1466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:43:16.935745 1466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.224.148:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.224.190:34877,127.10.224.189:39819,127.10.224.188:45239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/krb5kdc/kudu_127.10.224.188.keytab
--principal=kudu/127.10.224.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.224.188:45239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.224.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:43:16.936890 1466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:43:16.937927 1466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:43:16.947347 1471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:43:16.947360 1472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:16.947492 1466 server_base.cc:1034] running on GCE node
W20260926 12:43:16.947347 1474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:43:16.948318 1466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:43:16.949622 1466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:43:16.950836 1466 hybrid_clock.cc:648] HybridClock initialized: now 1790426596950786 us; error 72 us; skew 500 ppm
Sep 26 12:43:16 dist-test-slave-m3pj krb5kdc[1303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:43:16.954185 1466 init.cc:377] Logged in from keytab as kudu/127.10.224.188@KRBTEST.COM (short username kudu)
I20260926 12:43:16.955144 1466 webserver.cc:460] Webserver started at http://127.10.224.188:38045/ using document root <none> and password file <none>
I20260926 12:43:16.955804 1466 fs_manager.cc:359] Metadata directory not provided
I20260926 12:43:16.955880 1466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:43:16.956063 1466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:43:16.958228 1466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/data/instance:
uuid: "013af95c678e451dbdae652dd10b2f25"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "eeb0d243e20824ae25babaf715632882"
server_key_iv: "df9400cc189fe5d55ed4330f1abf7e0d"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.958860 1466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/wal/instance:
uuid: "013af95c678e451dbdae652dd10b2f25"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "eeb0d243e20824ae25babaf715632882"
server_key_iv: "df9400cc189fe5d55ed4330f1abf7e0d"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.962941 1466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:43:16.965754 1481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:43:16.967079 1466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:43:16.967221 1466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/data,/tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/wal
uuid: "013af95c678e451dbdae652dd10b2f25"
format_stamp: "Formatted at 2026-09-26 12:43:16 on dist-test-slave-m3pj"
server_key: "eeb0d243e20824ae25babaf715632882"
server_key_iv: "df9400cc189fe5d55ed4330f1abf7e0d"
server_key_version: "encryptionkey@0"
I20260926 12:43:16.967355 1466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:43:16.976604 1466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:43:16.980103 1466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:43:16.980322 1466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:43:16.988124 1466 rpc_server.cc:307] RPC server started. Bound to: 127.10.224.188:45239
I20260926 12:43:16.988441 1533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.224.188:45239 every 8 connection(s)
I20260926 12:43:16.989269 1466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/data/info.pb
I20260926 12:43:16.992497 1534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:43:16.994681 11138 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu as pid 1466
I20260926 12:43:16.994808 11138 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbzlsq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790426298078858-11138-0/minicluster-data/master-2/wal/instance
I20260926 12:43:16.995121 11138 external_mini_cluster.cc:1428] Setting key c49af869c8220e840f9090dd3f4902a8
I20260926 12:43:16.997360 1534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:17.002398 1384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } attempt: 3
Sep 26 12:43:17 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
Sep 26 12:43:17 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
Sep 26 12:43:17 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.190@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:43:17.012657 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.000073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.10.224.190:34877)
0926 12:43:17.000208 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.000214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.000708 (+ 494us) server_negotiation.cc:407] Connection header received
0926 12:43:17.000873 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.000875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.000919 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.000982 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.001961 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.002763 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.004007 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.004339 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.004411 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.008424 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.008428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.008431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.008460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.010152 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.010804 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.010810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.010812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.010899 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.011321 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.011325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.011327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.011660 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.011746 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.012298 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.012452 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:43:17.013391 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.002890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47460)
0926 12:43:17.003099 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:17.003118 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:17.003204 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:17.003755 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:17.003758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:17.003980 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:17.004084 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.004092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.004987 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.004992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.005783 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.005795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.006556 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.006564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.006689 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.007240 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:17.007260 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:17.008872 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:17.011823 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.011827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.011829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.012032 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.012323 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.012326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.012328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.012436 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.012886 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:17.012889 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:17.012988 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 12:43:17.013099 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 12:43:17.013249 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 12:43:17.014190 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.002947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.10.224.188:45239)
0926 12:43:17.003488 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.003496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.003507 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:43:17.003585 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.003588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.003636 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.003737 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.004210 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.004894 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.005914 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.006369 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.006937 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.009056 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.009060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.009063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.009096 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.011716 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.012141 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.012144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.012146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.012227 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.012554 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.012560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.012565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.012742 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.012894 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.013151 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.013272 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":186,"threads_started":1}
I20260926 12:43:17.014200 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.996759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.10.224.188:45239)
0926 12:43:16.997247 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:16.997268 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:16.997311 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:43:16.998201 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:16.998228 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:16.998696 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:16.999059 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:16.999953 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.001812 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.003092 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.003799 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.003969 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.007114 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.007121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.007134 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.007166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.009205 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.010532 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.010541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.010545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.010711 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.011202 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.011205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.011206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.011765 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.012655 (+ 890us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.013060 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.013369 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":339,"thread_start_us":171,"threads_started":1}
I20260926 12:43:17.014200 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:16.999912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.190:34877 (local address 127.0.0.1:48722)
0926 12:43:17.000559 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:17.000612 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:17.000757 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:17.001077 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:17.001083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:17.001584 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:17.001796 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.001810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.002913 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.002917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.003891 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.003899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.004606 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.004609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.004711 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.005312 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:17.005332 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:17.008249 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:17.010298 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.010310 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.010324 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.010688 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.011013 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.011018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.011021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.011195 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.011836 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:17.011843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:17.012004 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 12:43:17.012846 (+ 842us) client_negotiation.cc:240] Negotiation successful
0926 12:43:17.013230 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":133,"threads_started":1}
I20260926 12:43:17.016275 1534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:17.019086 1384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7: Bootstrap starting.
I20260926 12:43:17.019309 1459 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } attempt: 3
Sep 26 12:43:17 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
Sep 26 12:43:17 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.190@KRBTEST.COM
I20260926 12:43:17.023049 1384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:17.025513 1384 log.cc:824] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:43:17 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.189@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:43:17.027486 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.017072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.10.224.189:39819)
0926 12:43:17.017195 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.017200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.017259 (+ 59us) server_negotiation.cc:407] Connection header received
0926 12:43:17.017372 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.017383 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.017438 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.017531 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.018024 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.018941 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.020019 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.020332 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.020403 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.023300 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.023307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.023313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.023339 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.025453 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.025961 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.025965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.025967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.026066 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.026398 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.026402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.026404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.026639 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.026708 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.027151 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.027286 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 12:43:17.028050 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.016812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.189:39819 (local address 127.0.0.1:53498)
0926 12:43:17.017207 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:17.017221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:17.017292 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:17.017568 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:17.017571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:17.017750 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:17.017876 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.017884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.019069 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.019072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.019898 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.019907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.020567 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.020573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.020672 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.021207 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:17.021222 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:17.023133 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:17.025594 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.025599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.025603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.025846 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.026190 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.026196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.026201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.026296 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.026800 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:17.026803 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:17.026906 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 12:43:17.027693 (+ 787us) client_negotiation.cc:240] Negotiation successful
0926 12:43:17.027851 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317}
I20260926 12:43:17.028697 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.016899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.10.224.190:34877)
0926 12:43:17.017015 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.017019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.017070 (+ 51us) server_negotiation.cc:407] Connection header received
0926 12:43:17.017235 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.017241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.017294 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.017362 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.018206 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.018695 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.019771 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.020120 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.020187 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.023349 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.023353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.023355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.023381 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.025763 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.026471 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.026475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.026477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.026579 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.026966 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.026970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.026972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.027208 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.027389 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.028388 (+ 999us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.028534 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:43:17.028829 1534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:43:17.028996 1384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7: No bootstrap required, opened a new log
I20260926 12:43:17.030332 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.019741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47474)
0926 12:43:17.019989 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:17.020005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:17.020086 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:17.020679 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:17.020683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:17.020912 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:17.021031 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.021042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.022247 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.022250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.023117 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.023130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.023995 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.023998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.024106 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.024609 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:17.024628 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:17.026167 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:17.028656 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.028659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.028661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.028908 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.029231 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.029237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.029242 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.029334 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.029809 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:17.029813 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:17.029920 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 12:43:17.030018 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 12:43:17.030159 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260926 12:43:17.030951 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.020292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.10.224.188:45239)
0926 12:43:17.020418 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.020422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.020436 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:43:17.020472 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.020474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.020523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.021161 (+ 638us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.021290 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.022098 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.023403 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.023866 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.023992 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.026305 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.026309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.026312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.026338 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.028358 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.029041 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.029043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.029045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.029123 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.029446 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.029449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.029451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.029674 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.030633 (+ 959us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.030701 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.030805 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:43:17.033759 1459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047: Bootstrap starting.
Sep 26 12:43:17 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.224.188@KRBTEST.COM for kudu/127.10.224.188@KRBTEST.COM
I20260926 12:43:17.035107 1384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } }
I20260926 12:43:17.035374 1384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8360ad2e9b43d0bf8fd195031215d7, State: Initialized, Role: FOLLOWER
I20260926 12:43:17.036407 1459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:17.036468 1384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } }
I20260926 12:43:17.037256 1459 log.cc:824] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:17.037487 1541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.037757 1541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:17.038555 1384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:17.039336 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.029354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.224.188:45239 (local address 127.0.0.1:47484)
0926 12:43:17.029674 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 12:43:17.029687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:43:17.029837 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:43:17.030137 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:43:17.030140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:43:17.030295 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:43:17.030381 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.030387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.031366 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.031370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.032284 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:43:17.032296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.032968 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.032972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:43:17.033067 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.033641 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:43:17.033658 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:43:17.035349 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:43:17.037671 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.037674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.037677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.037926 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.038224 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:43:17.038227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:43:17.038229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:43:17.038310 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:43:17.038799 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:43:17.038802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:43:17.038888 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 12:43:17.039008 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 12:43:17.039152 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260926 12:43:17.039978 1459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047: No bootstrap required, opened a new log
I20260926 12:43:17.040046 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.029504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.10.224.188:45239)
0926 12:43:17.029616 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.029620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.029782 (+ 162us) server_negotiation.cc:407] Connection header received
0926 12:43:17.029925 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.029929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.029967 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.030042 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.030498 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.031251 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.032437 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.032830 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.032943 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.035547 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.035552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.035555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.035581 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.037541 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.038038 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.038042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.038044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.038127 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.038417 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.038420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.038422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.038663 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.038763 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.039743 (+ 980us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.039888 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 12:43:17 dist-test-slave-m3pj krb5kdc[1303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.224.189@KRBTEST.COM
I20260926 12:43:17.044525 1459 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } }
W20260926 12:43:17.044709 1553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:43:17.044771 1553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:43:17.044775 1459 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ecc90e38ec41aaa9466516a187a047, State: Initialized, Role: FOLLOWER
I20260926 12:43:17.045164 1459 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } }
I20260926 12:43:17.046746 1459 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:17.048679 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.034658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.10.224.189:39819)
0926 12:43:17.037769 (+ 3111us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.037776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.037792 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:43:17.037832 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.037837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.037886 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.038012 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.038893 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.039845 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.041132 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.041444 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.041529 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.044747 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.044751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.044753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.044780 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.046638 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.047207 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.047211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.047212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.047294 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.047654 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.047657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.047659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.047879 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.048024 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.048399 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.048533 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3035}
I20260926 12:43:17.050078 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25: Bootstrap starting.
I20260926 12:43:17.044499 1316 master_runner.cc:186] Error getting master registration for 127.10.224.189:39819: OK, instance_id { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" instance_seqno: 1790426596764000 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:43:17.051220 1316 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:43:17.052767 11138 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:43:17.052892 1534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25: Neither blocks nor log segments found. Creating new log.
I20260926 12:43:17.053398 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.053546 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:17.053795 1534 log.cc:824] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25: Log is configured to *not* fsync() on all Append() calls
I20260926 12:43:17.054309 1358 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9f8360ad2e9b43d0bf8fd195031215d7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48736
I20260926 12:43:17.054570 1358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 12:43:17.054708 1358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:17.055943 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25: No bootstrap required, opened a new log
I20260926 12:43:17.056712 1358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } }
I20260926 12:43:17.058369 1433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f9ecc90e38ec41aaa9466516a187a047"
I20260926 12:43:17.058624 1433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [term 0 FOLLOWER]: Advancing to term 1
W20260926 12:43:17.058823 1566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:43:17.058894 1566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:43:17.060182 1534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } }
I20260926 12:43:17.060437 1534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013af95c678e451dbdae652dd10b2f25, State: Initialized, Role: FOLLOWER
I20260926 12:43:17.060781 1534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } }
I20260926 12:43:17.061292 1569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.061420 1569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:17.061564 1433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f8360ad2e9b43d0bf8fd195031215d7 in term 1.
I20260926 12:43:17.062075 1534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:43:17.062831 1334 leader_election.cc:302] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f8360ad2e9b43d0bf8fd195031215d7, f9ecc90e38ec41aaa9466516a187a047; no voters:
I20260926 12:43:17.063359 1541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:43:17.063671 1541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [term 1 LEADER]: Becoming Leader. State: Replica: 9f8360ad2e9b43d0bf8fd195031215d7, State: Running, Role: LEADER
I20260926 12:43:17.063689 1358 leader_election.cc:288] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [CANDIDATE]: Term 1 election: Requested vote from peers f9ecc90e38ec41aaa9466516a187a047 (127.10.224.189:39819), 013af95c678e451dbdae652dd10b2f25 (127.10.224.188:45239)
I20260926 12:43:17.064023 1541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } }
I20260926 12:43:17.064091 1508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "013af95c678e451dbdae652dd10b2f25"
I20260926 12:43:17.064322 1508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:43:17.065893 1508 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f8360ad2e9b43d0bf8fd195031215d7 in term 1.
I20260926 12:43:17.066279 1575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8360ad2e9b43d0bf8fd195031215d7. Latest consensus state: current_term: 1 leader_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.066386 1575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:17.066792 1586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:43:17.067886 1586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 12:43:17.070276 1587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:43:17.070453 1587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:43:17.073345 1390 master_runner.cc:263] No leader master found from master f9ecc90e38ec41aaa9466516a187a047
I20260926 12:43:17.073418 1390 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:43:17.082161 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.070542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.10.224.188:45239)
0926 12:43:17.070678 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.070685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.070959 (+ 274us) server_negotiation.cc:407] Connection header received
0926 12:43:17.071168 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.071171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.071222 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.071277 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.072790 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.073736 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.075163 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.075573 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.075651 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.078085 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.078090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.078094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.078135 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.080116 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.080660 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.080664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.080666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.080755 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.081089 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.081093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.081095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.081331 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.081456 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.081828 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.081978 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:43:17.084867 1433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [term 1 FOLLOWER]: Refusing update from remote peer 9f8360ad2e9b43d0bf8fd195031215d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:43:17.084964 1508 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [term 1 FOLLOWER]: Refusing update from remote peer 9f8360ad2e9b43d0bf8fd195031215d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:43:17.085584 1575 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:43:17.085901 1593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:43:17.090808 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8360ad2e9b43d0bf8fd195031215d7. Latest consensus state: current_term: 1 leader_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.090957 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:17.091341 1569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8360ad2e9b43d0bf8fd195031215d7. Latest consensus state: current_term: 1 leader_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.091506 1569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:17.094115 1461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.071439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.10.224.189:39819)
0926 12:43:17.071607 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.071612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.076439 (+ 4827us) server_negotiation.cc:407] Connection header received
0926 12:43:17.076688 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.076691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.076740 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.076789 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.078680 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.079454 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.082638 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.082988 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.083081 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.088663 (+ 5582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.088668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.088671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.088716 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.090768 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.091776 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.091781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.091783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.091911 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.092271 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.092275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.092277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.092901 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.093042 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.093759 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.093929 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:43:17.096774 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:17.066915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.10.224.190:34877)
0926 12:43:17.067624 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.067629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.076230 (+ 8601us) server_negotiation.cc:407] Connection header received
0926 12:43:17.077688 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.077691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.077746 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.077796 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.080683 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.081453 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.083226 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:17.083603 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:17.083684 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:17.092182 (+ 8498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:43:17.092187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:43:17.092190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:43:17.092223 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:43:17.094034 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.094562 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.094566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.094568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.094661 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:43:17.094955 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:43:17.094959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:43:17.094960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:43:17.095191 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:43:17.095302 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:17.096486 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0926 12:43:17.096612 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625}
I20260926 12:43:17.101454 1593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.101605 1593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:17.101759 1589 mvcc.cc:205] Tried to move back new op lower bound from 7333587341618188288 to 7333587341578379264. Current Snapshot: MvccSnapshot[applied={T|T < 7333587341618188288}]
I20260926 12:43:17.103444 1569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.103606 1569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:17.104044 1541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.104193 1541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7 [sys.catalog]: This master's current role is: LEADER
I20260926 12:43:17.106755 1595 mvcc.cc:205] Tried to move back new op lower bound from 7333587341618188288 to 7333587341578379264. Current Snapshot: MvccSnapshot[applied={T|T < 7333587341618188288}]
I20260926 12:43:17.108204 1594 mvcc.cc:205] Tried to move back new op lower bound from 7333587341618188288 to 7333587341578379264. Current Snapshot: MvccSnapshot[applied={T|T < 7333587341618188288}]
I20260926 12:43:17.108794 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f8360ad2e9b43d0bf8fd195031215d7" member_type: VOTER last_known_addr { host: "127.10.224.190" port: 34877 } } peers { permanent_uuid: "f9ecc90e38ec41aaa9466516a187a047" member_type: VOTER last_known_addr { host: "127.10.224.189" port: 39819 } } peers { permanent_uuid: "013af95c678e451dbdae652dd10b2f25" member_type: VOTER last_known_addr { host: "127.10.224.188" port: 45239 } } }
I20260926 12:43:17.108927 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:43:17.111855 1586 catalog_manager.cc:1347] Generated new cluster ID: 0dd0d5c9904e48688c01b413ae54dc76
I20260926 12:43:17.111933 1586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:43:17.151674 1586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:43:17.153223 1586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:43:17.174369 1586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f8360ad2e9b43d0bf8fd195031215d7: Generated new TSK 0
I20260926 12:43:17.175372 1586 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:43:17.208784 1387 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.205529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.10.224.190:34877)
0926 12:43:17.205661 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.205666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.207596 (+ 1930us) server_negotiation.cc:407] Connection header received
0926 12:43:17.207818 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.207821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.207878 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.207934 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.208646 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48754: BlockingRecv error: recv got EOF from 127.0.0.1:48754 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 12:43:17.210659 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.206309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.10.224.189:39819)
0926 12:43:17.206431 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.206436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.209240 (+ 2804us) server_negotiation.cc:407] Connection header received
0926 12:43:17.209603 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.209607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.209667 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.209724 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.210530 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53520: BlockingRecv error: recv got EOF from 127.0.0.1:53520 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 12:43:17.211820 1539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.207014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.10.224.188:45239)
0926 12:43:17.207170 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.207175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.209978 (+ 2803us) server_negotiation.cc:407] Connection header received
0926 12:43:17.211043 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.211047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.211101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.211156 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.211704 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47508: BlockingRecv error: recv got EOF from 127.0.0.1:47508 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 12:43:17.216775 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.215078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.10.224.189:39819)
0926 12:43:17.215194 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.215198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.215772 (+ 574us) server_negotiation.cc:407] Connection header received
0926 12:43:17.215990 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.215993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.216083 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.216139 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.216658 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53524: BlockingRecv error: recv got EOF from 127.0.0.1:53524 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 12:43:17.222198 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.220567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.10.224.189:39819)
0926 12:43:17.220678 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.220683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.221240 (+ 557us) server_negotiation.cc:407] Connection header received
0926 12:43:17.221430 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.221434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.221491 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.221546 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.222075 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53528: BlockingRecv error: recv got EOF from 127.0.0.1:53528 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 12:43:17.228113 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.226365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.10.224.189:39819)
0926 12:43:17.226475 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.226480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.227069 (+ 589us) server_negotiation.cc:407] Connection header received
0926 12:43:17.227274 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.227277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.227334 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.227389 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.227976 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53544: BlockingRecv error: recv got EOF from 127.0.0.1:53544 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 12:43:17.236773 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.234686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.10.224.189:39819)
0926 12:43:17.234865 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.234870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.235592 (+ 722us) server_negotiation.cc:407] Connection header received
0926 12:43:17.235886 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.235890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.235952 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.236023 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.236648 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53550: BlockingRecv error: recv got EOF from 127.0.0.1:53550 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 12:43:17.249411 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.247428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.10.224.189:39819)
0926 12:43:17.247593 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.247599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.248261 (+ 662us) server_negotiation.cc:407] Connection header received
0926 12:43:17.248501 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.248505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.248570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.248631 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.249275 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53560: BlockingRecv error: recv got EOF from 127.0.0.1:53560 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 12:43:17.270718 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.268354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.10.224.189:39819)
0926 12:43:17.268512 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.268517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.269336 (+ 819us) server_negotiation.cc:407] Connection header received
0926 12:43:17.269608 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.269613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.269680 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.269745 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.270582 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53566: BlockingRecv error: recv got EOF from 127.0.0.1:53566 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 12:43:17.307350 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.305746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.10.224.189:39819)
0926 12:43:17.305896 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.305901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.306112 (+ 211us) server_negotiation.cc:407] Connection header received
0926 12:43:17.306252 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.306256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.306319 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.306429 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.307181 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53578: BlockingRecv error: recv got EOF from 127.0.0.1:53578 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 12:43:17.374907 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.373649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.10.224.189:39819)
0926 12:43:17.373829 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.373833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.373909 (+ 76us) server_negotiation.cc:407] Connection header received
0926 12:43:17.374047 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.374050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.374101 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.374184 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.374749 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53580: BlockingRecv error: recv got EOF from 127.0.0.1:53580 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 12:43:17.506315 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.505179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.10.224.189:39819)
0926 12:43:17.505317 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.505322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.505432 (+ 110us) server_negotiation.cc:407] Connection header received
0926 12:43:17.505551 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.505553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.505602 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.505697 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.506179 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53586: BlockingRecv error: recv got EOF from 127.0.0.1:53586 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 12:43:17.765970 1461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:43:17.764652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53594 (local address 127.10.224.189:39819)
0926 12:43:17.764797 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:17.764801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:17.765055 (+ 254us) server_negotiation.cc:407] Connection header received
0926 12:43:17.765168 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:17.765171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:17.765227 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:17.765306 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:43:17.765840 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53594: BlockingRecv error: recv got EOF from 127.0.0.1:53594 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:43:18.060271 1566 catalog_manager.cc:1259] Loaded cluster ID: 0dd0d5c9904e48688c01b413ae54dc76
I20260926 12:43:18.060355 1566 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047: loading cluster ID for follower catalog manager: success
I20260926 12:43:18.062098 1566 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047: acquiring CA information for follower catalog manager: success
I20260926 12:43:18.063494 1566 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9ecc90e38ec41aaa9466516a187a047: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:43:18.072067 1587 catalog_manager.cc:1259] Loaded cluster ID: 0dd0d5c9904e48688c01b413ae54dc76
I20260926 12:43:18.072145 1587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25: loading cluster ID for follower catalog manager: success
I20260926 12:43:18.073796 1587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25: acquiring CA information for follower catalog manager: success
I20260926 12:43:18.074889 1587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 013af95c678e451dbdae652dd10b2f25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:43:18.284350 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:18.280339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.10.224.189:39819)
0926 12:43:18.280591 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:18.280594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:18.280611 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:43:18.280762 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:18.280765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:18.280892 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:18.280990 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:18.281252 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.281815 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.282800 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.283043 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.283125 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:18.283476 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:18.283938 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:18.284039 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:18.284160 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 12:43:18.284215 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
I20260926 12:43:18.291671 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:18.287649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.10.224.188:45239)
0926 12:43:18.287874 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:18.287877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:18.287890 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:43:18.288044 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:18.288047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:18.288165 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:18.288262 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:18.288558 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.289132 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.289970 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.290220 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.290304 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:18.290690 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:18.291159 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:18.291265 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:18.291371 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 12:43:18.291426 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
I20260926 12:43:18.299139 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:18.295026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.10.224.188:45239)
0926 12:43:18.295126 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:18.295130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:18.295669 (+ 539us) server_negotiation.cc:407] Connection header received
0926 12:43:18.295744 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:18.295749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:18.295811 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:18.295873 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:18.296212 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.296971 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.298013 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.298275 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.298369 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:18.298692 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:18.298765 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:18.298843 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:18.298941 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 12:43:18.298990 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:43:18.299582 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:18.295338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.10.224.189:39819)
0926 12:43:18.295490 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:18.295494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:18.295936 (+ 442us) server_negotiation.cc:407] Connection header received
0926 12:43:18.296153 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:18.296156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:18.296216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:18.296284 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:18.296604 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.297290 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.298302 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.298616 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.298691 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:18.299057 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:18.299142 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:18.299188 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:18.299351 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 12:43:18.299412 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:43:18.299642 1677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:43:18.294872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.10.224.190:34877)
0926 12:43:18.295141 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 12:43:18.295146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:43:18.295360 (+ 214us) server_negotiation.cc:407] Connection header received
0926 12:43:18.295488 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:43:18.295491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:43:18.295702 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:43:18.295827 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:43:18.296080 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.296667 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.297809 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:43:18.298170 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:43:18.298293 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:43:18.298605 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:43:18.298975 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:43:18.299056 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:43:18.299414 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 12:43:18.299474 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260926 12:43:18.307915 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 1316
I20260926 12:43:18.314706 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 1390
I20260926 12:43:18.320750 11138 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbzlsq/build/debug/bin/kudu with pid 1466
2026-09-26T12:43:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3175 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6367 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (300234 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST